tag:blogger.com,1999:blog-20277096212505021662024-03-19T14:48:48.430+05:30Technology blog by Rathish kumarPersonal blog on Software Architecture, Cloud Computing, Data Engineering and Analytics.Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.comBlogger50125tag:blogger.com,1999:blog-2027709621250502166.post-92023510634283671502023-10-27T15:43:00.002+05:302023-10-27T15:43:57.225+05:30How to perform join operation in BigQuery? Exploring BigQuery Join Operations: Broadcast and Hashing Joins & Nested and Repeated Structures.<p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiOJm6kcq2xgV4xcUsx85c1vLQ-WDJnq1vakIjOXxqI34ql5HUe_AymnB-2Jg9YGux5g90JJXdGOSbm1TJ6PU86WzICFmI-z-wbKmlQ3JZ0iuYHQ9hMfyaUIY_iVosBn9cJkuTSeD74vXXkTYN2cMim3_3UebdGIFXYuRZeoicyQDxUIXAPgJW-9aSMcg" style="margin-left: auto; margin-right: auto;"><img alt="BigQuery - SQL Joins" data-original-height="2500" data-original-width="2000" src="https://blogger.googleusercontent.com/img/a/AVvXsEiOJm6kcq2xgV4xcUsx85c1vLQ-WDJnq1vakIjOXxqI34ql5HUe_AymnB-2Jg9YGux5g90JJXdGOSbm1TJ6PU86WzICFmI-z-wbKmlQ3JZ0iuYHQ9hMfyaUIY_iVosBn9cJkuTSeD74vXXkTYN2cMim3_3UebdGIFXYuRZeoicyQDxUIXAPgJW-9aSMcg=s16000" title="BigQuery - SQL Joins" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">BigQuery - SQL Joins (Photo by <a href="https://unsplash.com/@resourcedatabase?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Resource Database</a> on <a href="https://unsplash.com/photos/I1tek-IwO3w?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>) </td></tr></tbody></table><p><span style="font-family: Noto Sans; font-size: x-large;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: x-large;">S</span><span style="font-family: Noto Sans; font-size: medium;">QL joins are used to combine columns from multiple tables to get desired result set. In a typical Relational model we use normalized tables, each table represents an entity (example: employee, department, etc) and its relationships and when we need to get data from more than one tables, for example employee name and employee department, we use joins to combine employee name column from employee table, department name column from department table based on employee number key column, which is available on both the tables.</span></p><p><span></span></p><a name='more'></a><span style="font-family: Noto Sans; font-size: medium;"><br /></span><p></p><p><span style="font-family: Noto Sans; font-size: medium;">Similarly, typical data warehouse setup follows Star or Snowflake schema consisting of a primary fact table and satellite dimension tables. Fact tables represents events (example: orders table in a ecommerce business) and dimension table represents attributes and slowly changing information (example: customer, product tables).</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhiSkFk_KUeRNsJbrte42DNmjUBN06H0xYJgbI62qhOVJLM8_pUAS9lHllvS_d1Uhfu3-cb9NOe9zNCsFc0xcH652T4D1zlvfQt-4ifi7LhZV6qn2S5n2VM8F2hpI_ungUvb3zboZZojF63n879AkVa_ENmi9ELTVgEvSRc3TgLGSdAs5MomlwWET9YrA" style="margin-left: auto; margin-right: auto;"><img alt="" data-original-height="430" data-original-width="710" height="194" src="https://blogger.googleusercontent.com/img/a/AVvXsEhiSkFk_KUeRNsJbrte42DNmjUBN06H0xYJgbI62qhOVJLM8_pUAS9lHllvS_d1Uhfu3-cb9NOe9zNCsFc0xcH652T4D1zlvfQt-4ifi7LhZV6qn2S5n2VM8F2hpI_ungUvb3zboZZojF63n879AkVa_ENmi9ELTVgEvSRc3TgLGSdAs5MomlwWET9YrA" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Star Schema Example (Source: <a href="https://commons.wikimedia.org/w/index.php?curid=76901169" rel="nofollow" target="_blank">Wikipedia by SqlPac</a>)</td></tr></tbody></table><br /><br /><p></p><p><span style="font-family: Noto Sans; font-size: medium;">Fact tables are denormalized for faster reads and dimension tables are normalized. In most cases, your analytics query need to join a fact table with multiple dimension tables to get the desired results, for example finding top 10 customers whose transaction value is greater than $100 and their ordered products. These type of queries requires join operation on transaction, customer and product tables to get desired result sets.</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;">BigQuery as a serverless enterprise data warehouse support ANSI SQL joins types such as inner join, left outer join, right outer join, full outer join and cross join. Let's discuss about how join works in BigQuery, basic join types and try it out on BigQuery public dataset: <span style="background-color: #fffffe; color: #0d904f; white-space: pre;">`bigquery-public-data.thelook_ecommerce`</span></span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><h1 style="text-align: left;"><span style="font-family: Noto Sans;">How join operations are performed in BigQuery?</span></h1><p></p><p style="text-align: left;"></p><ul style="text-align: left;"><li><span style="font-family: Noto Sans; font-size: medium;">Create two result set items: In BigQuery join operation can be performed on tables, subquery, Arrays or WITH statement.</span></li></ul><ul style="text-align: left;"><li><span style="font-family: Noto Sans; font-size: medium;">Identify a join condition: The join condition does not need to be an equality condition; any Boolean condition can be used. The join condition specified using ON clause or USING clause.</span></li></ul><ul style="text-align: left;"><li><span style="font-family: Noto Sans; font-size: medium;">Select the column list you wanted in final result set: If the joining result set/table consists of identical column names, use alias to differentiate the columns.</span></li></ul><ul style="text-align: left;"><li><span style="font-family: Noto Sans; font-size: medium;">Specify the join type: if no joining type is specified, by default considered as inner join.</span></li></ul><p></p><p></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Inner Join</span></h2><p><span style="font-family: Noto Sans; font-size: medium;">Inner Join returns only rows that meets the join condition. Effectively calculates Cartesian product of two tables based on a common values and discard all rows that do not meet the join condition.</span></p><p><br /></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEigwFFUY1WaancOa9Gyi9b00MDZKTKefJpUWuD4epqMe40GJkEfLEuxGFd9LHEfklWhbp_sPvGun8kc_gXftSVe92K2XRim7b76TMze2nbpooTwGLtbX8b0n_pW1ln7X3sYgbe7PNRSYPy865NWSbbBAzgsE-fnbcRjY9_dNdajby_E71C2qu610AcarQ" style="margin-left: auto; margin-right: auto;"><img alt="BigQuery - SQL Inner Join" data-original-height="1194" data-original-width="1852" src="https://blogger.googleusercontent.com/img/a/AVvXsEigwFFUY1WaancOa9Gyi9b00MDZKTKefJpUWuD4epqMe40GJkEfLEuxGFd9LHEfklWhbp_sPvGun8kc_gXftSVe92K2XRim7b76TMze2nbpooTwGLtbX8b0n_pW1ln7X3sYgbe7PNRSYPy865NWSbbBAzgsE-fnbcRjY9_dNdajby_E71C2qu610AcarQ=s16000" title="BigQuery - SQL Inner Join" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">BigQuery - SQL Inner Join</td></tr></tbody></table><br /><span style="font-family: "Noto Sans"; font-size: large;">Joining table A and B and joining B and A are same. If the column names are same, join condition can be specified using USING clause.</span><p></p><p><br /></p><h3 style="text-align: left;"><span style="font-family: "Noto Sans";"><span style="font-size: medium;">Example:</span></span></h3><p><span style="font-family: Noto Sans; font-size: medium;">List product names and its order status.</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;">Product schema:</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div><span style="color: #c0caf5;"><br /></span></div><div><span style="color: #c0caf5;">❯</span> <span style="color: #9ece6a;">bq</span> <span style="color: #9ece6a;">query</span> <span style="color: #e0af68;">--nouse_legacy_sql</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">SELECT column_name, data_type, is_nullable FROM `bigquery-public-data.thelook_ecommerce.INFORMATION_SCHEMA.COLUMNS` </span></div><div><span style="color: #9ece6a;">WHERE table_name = "products";</span><span style="color: #89ddff;">'</span></div><br /><br /><div><span style="color: #c0caf5;">+------------------------+-----------+-------------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">column_name</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">data_type</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">is_nullable</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+------------------------+-----------+-------------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">id</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">INT64</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">cost</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">FLOAT64</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">category</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">STRING</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">name</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">STRING</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">brand</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">STRING</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">retail_price</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">FLOAT64</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">department</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">STRING</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">sku</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">STRING</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">distribution_center_id</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">INT64</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+------------------------+-----------+-------------+</span></div><div><span style="color: #c0caf5;"><br /></span></div></div></pre><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;">Order Items schema:</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #c0caf5;"><br /></span></div><div><span style="color: #c0caf5;">❯</span> <span style="color: #9ece6a;">bq</span> <span style="color: #9ece6a;">query</span> <span style="color: #e0af68;">--nouse_legacy_sql</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">SELECT column_name, data_type, is_nullable FROM `bigquery-public-data.thelook_ecommerce.INFORMATION_SCHEMA.COLUMNS`</span></div><div><span style="color: #9ece6a;">WHERE table_name = "order_items";</span><span style="color: #89ddff;">'</span></div><div><span style="color: #89ddff;"><br /></span></div><div><span style="color: #89ddff;"><br /></span></div><div><span style="color: #c0caf5;">+-------------------+-----------+-------------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">column_name</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">data_type</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">is_nullable</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+-------------------+-----------+-------------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">id</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">INT64</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">order_id</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">INT64</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">user_id</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">INT64</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">product_id</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">INT64</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">inventory_item_id</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">INT64</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">status</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">STRING</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">created_at</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">TIMESTAMP</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">shipped_at</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">TIMESTAMP</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">delivered_at</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">TIMESTAMP</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">returned_at</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">TIMESTAMP</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">sale_price</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">FLOAT64</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">YES</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+-------------------+-----------+-------------+</span></div><div><span style="color: #c0caf5;"><br /></span></div></div></div></pre><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;">Selecting Product name and Order status using inner join:</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><br /></span></div><div><div style="line-height: 20px;"><div> <span style="color: #7dcfff;">SELECT</span> </div><div> <span> </span><span style="color: #e0af68;">p</span>.<span style="color: #e0af68;">name as `Product Name`</span>,</div><div> <span> </span><span style="color: #e0af68;">oi</span>.<span style="color: #e0af68;">status as `Order Status`</span></div><div> <span style="color: #7dcfff;">FROM</span> </div><div><span> </span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.order_items</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> oi</div><div> <span style="color: #7dcfff;">INNER JOIN</span> </div><div> <span> </span><span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.products</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> p</div><div> <span style="color: #89ddff;">ON</span></div><div> <span> </span><span style="color: #e0af68;">oi</span>.<span style="color: #e0af68;">product_id</span> <span style="color: #bb9af7;">=</span> <span style="color: #e0af68;">p</span>.<span style="color: #e0af68;">id</span></div><div> <span style="color: #7dcfff;">LIMIT</span> <span style="color: #ff9e64;">10</span>;</div><div><br /></div><div><div style="line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #c0caf5;">+--------------------------------------------------------------------------------------------------------+--------------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Product</span> <span style="color: #9ece6a;">Name</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Order</span> <span style="color: #9ece6a;">Status</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+--------------------------------------------------------------------------------------------------------+--------------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Elegant</span> <span style="color: #9ece6a;">PASHMINA</span> <span style="color: #9ece6a;">SCARF</span> <span style="color: #9ece6a;">WRAP</span> <span style="color: #9ece6a;">SHAWL</span> <span style="color: #9ece6a;">STOLE</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Shipped</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Elegant</span> <span style="color: #9ece6a;">PASHMINA</span> <span style="color: #9ece6a;">SCARF</span> <span style="color: #9ece6a;">WRAP</span> <span style="color: #9ece6a;">SHAWL</span> <span style="color: #9ece6a;">STOLE</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Shipped</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Elegant</span> <span style="color: #9ece6a;">PASHMINA</span> <span style="color: #9ece6a;">SCARF</span> <span style="color: #9ece6a;">WRAP</span> <span style="color: #9ece6a;">SHAWL</span> <span style="color: #9ece6a;">STOLE</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Complete</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Elegant</span> <span style="color: #9ece6a;">PASHMINA</span> <span style="color: #9ece6a;">SCARF</span> <span style="color: #9ece6a;">WRAP</span> <span style="color: #9ece6a;">SHAWL</span> <span style="color: #9ece6a;">STOLE</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Complete</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Elegant</span> <span style="color: #9ece6a;">PASHMINA</span> <span style="color: #9ece6a;">SCARF</span> <span style="color: #9ece6a;">WRAP</span> <span style="color: #9ece6a;">SHAWL</span> <span style="color: #9ece6a;">STOLE</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Returned</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Elegant</span> <span style="color: #9ece6a;">PASHMINA</span> <span style="color: #9ece6a;">SCARF</span> <span style="color: #9ece6a;">WRAP</span> <span style="color: #9ece6a;">SHAWL</span> <span style="color: #9ece6a;">STOLE</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Returned</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Nice</span> <span style="color: #9ece6a;">Shades</span> <span style="color: #9ece6a;">Black</span> <span style="color: #9ece6a;">One</span> <span style="color: #9ece6a;">Size</span> <span style="color: #9ece6a;">Canvas</span> <span style="color: #9ece6a;">Military</span> <span style="color: #9ece6a;">Web</span> <span style="color: #9ece6a;">Belt</span> <span style="color: #9ece6a;">With</span> <span style="color: #9ece6a;">Black</span> <span style="color: #9ece6a;">Slider</span> <span style="color: #9ece6a;">Buckle.</span> <span style="color: #9ece6a;">Many</span> <span style="color: #9ece6a;">Colors</span> <span style="color: #9ece6a;">Available</span> <span style="color: #ff9e64;">56</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Complete</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Nice</span> <span style="color: #9ece6a;">Shades</span> <span style="color: #9ece6a;">Black</span> <span style="color: #9ece6a;">One</span> <span style="color: #9ece6a;">Size</span> <span style="color: #9ece6a;">Canvas</span> <span style="color: #9ece6a;">Military</span> <span style="color: #9ece6a;">Web</span> <span style="color: #9ece6a;">Belt</span> <span style="color: #9ece6a;">With</span> <span style="color: #9ece6a;">Black</span> <span style="color: #9ece6a;">Slider</span> <span style="color: #9ece6a;">Buckle.</span> <span style="color: #9ece6a;">Many</span> <span style="color: #9ece6a;">Colors</span> <span style="color: #9ece6a;">Available</span> <span style="color: #ff9e64;">56</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Complete</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Nice</span> <span style="color: #9ece6a;">Shades</span> <span style="color: #9ece6a;">Black</span> <span style="color: #9ece6a;">One</span> <span style="color: #9ece6a;">Size</span> <span style="color: #9ece6a;">Canvas</span> <span style="color: #9ece6a;">Military</span> <span style="color: #9ece6a;">Web</span> <span style="color: #9ece6a;">Belt</span> <span style="color: #9ece6a;">With</span> <span style="color: #9ece6a;">Black</span> <span style="color: #9ece6a;">Slider</span> <span style="color: #9ece6a;">Buckle.</span> <span style="color: #9ece6a;">Many</span> <span style="color: #9ece6a;">Colors</span> <span style="color: #9ece6a;">Available</span> <span style="color: #ff9e64;">56</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Returned</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Nice</span> <span style="color: #9ece6a;">Shades</span> <span style="color: #9ece6a;">Black</span> <span style="color: #9ece6a;">One</span> <span style="color: #9ece6a;">Size</span> <span style="color: #9ece6a;">Canvas</span> <span style="color: #9ece6a;">Military</span> <span style="color: #9ece6a;">Web</span> <span style="color: #9ece6a;">Belt</span> <span style="color: #9ece6a;">With</span> <span style="color: #9ece6a;">Black</span> <span style="color: #9ece6a;">Slider</span> <span style="color: #9ece6a;">Buckle.</span> <span style="color: #9ece6a;">Many</span> <span style="color: #9ece6a;">Colors</span> <span style="color: #9ece6a;">Available</span> <span style="color: #ff9e64;">56</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Returned</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+--------------------------------------------------------------------------------------------------------+--------------+</span></div><br /></div></div></div></div></div></div></div></pre><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Left Outer Join</span></h2><p><span style="font-family: Noto Sans; font-size: medium;">Returns all rows from the left side of the join even if the rows not matching join predicate and only matching rows from the right side of the join. All the remaining rows from the right side of join returned as NULL.</span></p><p></p><div class="separator" style="clear: both; text-align: center;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgD-zWZAmsKQyYgyRClg80hOZzMPak5_g3G3073Ju--mC52nMDI4ezwoGXmlpaLOhSaWQC_czUaXQ8SjpUM2tWZcNivLidKz6Y30oQjpLkjSmpttZBNlJFSTSGllgZJdGVgYAZdNuJ3lFZ1BfNlJiBAbmjwBxb1xfcHzxfDl9lbdJd3l1dpDZG9k5xYZw" style="margin-left: auto; margin-right: auto;"><img alt="BigQuery - SQL Left Join" data-original-height="756" data-original-width="1846" src="https://blogger.googleusercontent.com/img/a/AVvXsEgD-zWZAmsKQyYgyRClg80hOZzMPak5_g3G3073Ju--mC52nMDI4ezwoGXmlpaLOhSaWQC_czUaXQ8SjpUM2tWZcNivLidKz6Y30oQjpLkjSmpttZBNlJFSTSGllgZJdGVgYAZdNuJ3lFZ1BfNlJiBAbmjwBxb1xfcHzxfDl9lbdJd3l1dpDZG9k5xYZw=s16000" title="BigQuery - SQL Left Join" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">BigQuery - SQL Left Join</td></tr></tbody></table></div><p></p><h3 style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;"><br /></span></h3><h3 style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;">Example</span></h3><p><span style="font-family: Noto Sans; font-size: medium;">List all the products which never ordered (products not ordered even once).</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br /></span></div><div><div style="line-height: 20px;"><div style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><div style="line-height: 20px;"><div style="color: #a9b1d6;"><div style="line-height: 20px;"><div> <span style="color: #7dcfff;">SELECT</span> </div><div> <span style="color: #e0af68;">p</span>.<span style="color: #e0af68;">name</span> <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">Product Name</span><span style="color: #89ddff;">`</span>,</div><div> <span style="color: #e0af68;">oi</span>.<span style="color: #e0af68;">order_id</span> <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">Order ID</span><span style="color: #89ddff;">`</span></div><div> <span style="color: #7dcfff;">FROM</span> </div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.products</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> p</div><div> <span style="color: #7dcfff;">LEFT JOIN</span> </div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.order_items</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> oi</div><div> <span style="color: #89ddff;">ON</span></div><div> <span style="color: #e0af68;">oi</span>.<span style="color: #e0af68;">product_id</span> <span style="color: #bb9af7;">=</span> <span style="color: #e0af68;">p</span>.<span style="color: #e0af68;">id</span></div><div> <span style="color: #7dcfff;">WHERE</span> <span style="color: #e0af68;">oi</span>.<span style="color: #e0af68;">status</span> <span style="color: #89ddff;">IS</span> <span style="color: #89ddff;">NULL</span></div><div> <span style="color: #7dcfff;">LIMIT</span> <span style="color: #ff9e64;">10</span>; </div><div><br /></div><div><div style="line-height: 20px;"><div><span style="color: #c0caf5;">+-------------------------------------------------------------------------------------------+----------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Product</span> <span style="color: #9ece6a;">Name</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Order</span> <span style="color: #9ece6a;">ID</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+-------------------------------------------------------------------------------------------+----------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Cloudveil</span> <span style="color: #9ece6a;">Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Kahuna Short | NULL |</span></div><div><span style="color: #9ece6a;">| 30 inch Inseam Smith</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s</span> <span style="color: #9ece6a;">Denim</span> <span style="color: #9ece6a;">Fleece</span> <span style="color: #9ece6a;">-</span> <span style="color: #9ece6a;">lined</span> <span style="color: #9ece6a;">Jeans</span> <span style="color: #9ece6a;">Dark</span> <span style="color: #9ece6a;">Stonewash</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">RUDE</span> <span style="color: #9ece6a;">Dark</span> <span style="color: #9ece6a;">Vintage</span> <span style="color: #9ece6a;">Skinny</span> <span style="color: #9ece6a;">Jeans</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Nike</span> <span style="color: #9ece6a;">Golf</span> <span style="color: #9ece6a;">Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Flat Front Tech Short | NULL |</span></div><div><span style="color: #9ece6a;">| Hurley Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s</span> <span style="color: #9ece6a;">Poolside</span> <span style="color: #9ece6a;">Boardwalk</span> <span style="color: #9ece6a;">Walkshort</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Polo</span> <span style="color: #9ece6a;">Ralph</span> <span style="color: #9ece6a;">Lauren</span> <span style="color: #9ece6a;">Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Walk Golf Shorts Red | NULL |</span></div><div><span style="color: #9ece6a;">| Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s</span> <span style="color: #9ece6a;">Cashmere</span> <span style="color: #9ece6a;">Cardigan</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Lucky</span> <span style="color: #9ece6a;">Brand</span> <span style="color: #9ece6a;">Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Boxer Set | NULL |</span></div><div><span style="color: #9ece6a;">| ililily Simple Basic 100% Cotton Baseball 3/4 raglan sleeve T-shirt for Men (tshirts-008) | NULL |</span></div><div><span style="color: #9ece6a;">| Pendleton Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s</span> <span style="color: #9ece6a;">Classic</span> <span style="color: #9ece6a;">Fit</span> <span style="color: #9ece6a;">Board</span> <span style="color: #9ece6a;">Shirt</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+-------------------------------------------------------------------------------------------+----------+</span></div></div></div></div></div></div></div><div><div style="line-height: 20px;"><div><br /></div></div></div></div></div></div></div></pre><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;">The above query can rewritten using subqueries with EXISTS condition to fetch the results without using join as we just need the Product name, however we will continue with this syntax for explaining joins.</span></p><h2 style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;"><br /></span></h2><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Right Outer Join</span></h2><p><span style="font-family: "Noto Sans";"><span style="font-size: medium;">Returns all rows from the right side of join even if the rows are not matching the join predicate and returns only matching rows from the left side of the join. All the remaining rows from left side of the side returned as NULL.</span></span></p><p><span style="font-family: "Noto Sans";"><span style="font-size: medium;"><br /></span></span></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiyL3YH1X9sxiSWqVGeEgIkXnKsqxYjBieCVhDzYEinlsRIKtYFZLvJes47uR85lh_564zJe-csAwx1JnsJ0aBWAsrB2zbfK6vcdUPbXvqvWl_Xdg-JBHuuT0mA3muf4HEBQ3l842qPr5k0PSmLJynmVIsjolENF4dIr9ZpnOhYFX4ym7IQUMmqMjMbXw" style="margin-left: auto; margin-right: auto;"><img alt="BigQuery - SQL Right Join" data-original-height="758" data-original-width="1860" src="https://blogger.googleusercontent.com/img/a/AVvXsEiyL3YH1X9sxiSWqVGeEgIkXnKsqxYjBieCVhDzYEinlsRIKtYFZLvJes47uR85lh_564zJe-csAwx1JnsJ0aBWAsrB2zbfK6vcdUPbXvqvWl_Xdg-JBHuuT0mA3muf4HEBQ3l842qPr5k0PSmLJynmVIsjolENF4dIr9ZpnOhYFX4ym7IQUMmqMjMbXw=s16000" title="BigQuery - SQL Right Join" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">BigQuery - SQL Right Join</td></tr></tbody></table><br /><p></p><h3 style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;"><br /></span></h3><h3 style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;">Example</span></h3><p><span style="font-family: "Noto Sans";"><span style="font-size: medium;">List all the orders with no products (where products removed from the product table, but there were previous orders exists for those products).</span></span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><br /></span></div><div><div style="line-height: 20px;"><div style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><div style="line-height: 20px;"><div style="color: #a9b1d6;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div> <span style="color: #7dcfff;">SELECT</span> </div><div> <span style="color: #e0af68;">p</span>.<span style="color: #e0af68;">name</span> <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">Product Name</span><span style="color: #89ddff;">`</span>,</div><div> <span style="color: #e0af68;">oi</span>.<span style="color: #e0af68;">order_id</span> <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">Order ID</span><span style="color: #89ddff;">`</span>,</div><div> <span style="color: #7dcfff;">FROM</span> </div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.products</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> p</div><div> <span style="color: #7dcfff;">RIGHT JOIN</span> </div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.order_items</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> oi</div><div> <span style="color: #89ddff;">ON</span></div><div> <span style="color: #e0af68;">oi</span>.<span style="color: #e0af68;">product_id</span> <span style="color: #bb9af7;">=</span> <span style="color: #e0af68;">p</span>.<span style="color: #e0af68;">id</span></div><div> <span style="color: #7dcfff;">WHERE</span> <span style="color: #e0af68;">p</span>.<span style="color: #e0af68;">name</span> <span style="color: #89ddff;">IS</span> <span style="color: #89ddff;">NULL</span></div></div><div> <span style="color: #7dcfff;">LIMIT</span> <span style="color: #ff9e64;">10</span>; </div><div><br /></div><div><div style="line-height: 20px;"><div><div style="line-height: 20px;"><div><span style="color: #c0caf5;">+--------------+----------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Product</span> <span style="color: #9ece6a;">Name</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Order</span> <span style="color: #9ece6a;">ID</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+--------------+----------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">39411</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">75248</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">76724</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">117507</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">32556</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">42489</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">57019</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">92339</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">49356</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">75177</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+--------------+----------+</span></div></div></div></div></div></div></div></div></div></div></div></div></div></pre><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Full Outer Join</span></h2><p><span style="font-family: Noto Sans; font-size: medium;">Full join returns all rows from both side of join for matching conditions and If a row from one side does not join any other row in other side, the row returns with NULL value for other side of the table.</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgbtaCVK_-j2ZlG3Gez2kUYqt8N-vP2IpHHeQLK7ugOSdqiBmOKQUgJYVhlHAe5HYEptOqF_-1RpzKF5F_m6I0tz-S3G6sPZXvuB_WDSuoXxoLhenowV6hHU3rQX73_w7YcNe9d5YRvYv8Y5LhBUE-2xbqqpxyDxR_UX8Po116mYnVwLm7S8BHZVrZKzA" style="margin-left: auto; margin-right: auto;"><img alt="BigQuery - SQL Full Outer Join" data-original-height="746" data-original-width="1876" src="https://blogger.googleusercontent.com/img/a/AVvXsEgbtaCVK_-j2ZlG3Gez2kUYqt8N-vP2IpHHeQLK7ugOSdqiBmOKQUgJYVhlHAe5HYEptOqF_-1RpzKF5F_m6I0tz-S3G6sPZXvuB_WDSuoXxoLhenowV6hHU3rQX73_w7YcNe9d5YRvYv8Y5LhBUE-2xbqqpxyDxR_UX8Po116mYnVwLm7S8BHZVrZKzA=s16000" title="BigQuery - SQL Full Outer Join" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">BigQuery - SQL Full Outer Join</td></tr></tbody></table><br /><br /><p></p><h3 style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;"><br /></span></h3><h3 style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;">Example</span></h3><div><span style="font-family: Noto Sans; font-size: medium;">List all products never ordered and orders with no products</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br /></span></div><div><div style="line-height: 20px;"><div style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><div style="line-height: 20px;"><div style="color: #a9b1d6;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div> <span style="color: #7dcfff;">SELECT</span> </div><div> <span style="color: #e0af68;">p</span>.<span style="color: #e0af68;">name</span> <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">Product Name</span><span style="color: #89ddff;">`</span>,</div><div> <span style="color: #e0af68;">oi</span>.<span style="color: #e0af68;">order_id</span> <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">Order ID</span><span style="color: #89ddff;">`</span>,</div><div> <span style="color: #7dcfff;">FROM</span> </div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.products</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> p</div><div> <span style="color: #7dcfff;">FULL OUTER JOIN</span> </div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.order_items</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> oi</div><div> <span style="color: #89ddff;">ON</span></div><div> <span style="color: #e0af68;">oi</span>.<span style="color: #e0af68;">product_id</span> <span style="color: #bb9af7;">=</span> <span style="color: #e0af68;">p</span>.<span style="color: #e0af68;">id</span></div></div><div> <span style="color: #7dcfff;">WHERE</span> <span style="color: #e0af68;">p</span>.<span style="color: #e0af68;">name</span> <span style="color: #89ddff;">IS</span> <span style="color: #89ddff;">NULL</span> <span style="color: #7dcfff;">OR</span> <span style="color: #e0af68;">oi</span>.<span style="color: #e0af68;">order_id</span> <span style="color: #89ddff;">IS</span> <span style="color: #89ddff;">NULL</span>; </div></div><div><br /></div><div><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #c0caf5;">+------------------------------------------------------------------------------------------------------------------------+----------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Product</span> <span style="color: #9ece6a;">Name</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Order</span> <span style="color: #9ece6a;">ID</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+------------------------------------------------------------------------------------------------------------------------+----------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Lucky</span> <span style="color: #9ece6a;">Brand</span> <span style="color: #9ece6a;">Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Boxer Set | NULL |</span></div><div><span style="color: #9ece6a;">| NULL | 39411 |</span></div><div><span style="color: #9ece6a;">| NULL | 75248 |</span></div><div><span style="color: #9ece6a;">| NULL | 76724 |</span></div><div><span style="color: #9ece6a;">| NULL | 117507 |</span></div><div><span style="color: #9ece6a;">| NULL | 32556 |</span></div><div><span style="color: #9ece6a;">| NULL | 42489 |</span></div><div><span style="color: #9ece6a;">| NULL | 57019 |</span></div><div><span style="color: #9ece6a;">| NULL | 92339 |</span></div><div><span style="color: #9ece6a;">| So So Happy Junior Ozzie Hoodie in Blue | NULL |</span></div><div><span style="color: #9ece6a;">| ililily Simple Basic 100% Cotton Baseball 3/4 raglan sleeve T-shirt for Men (tshirts-008) | NULL |</span></div><div><span style="color: #9ece6a;">| Cloudveil Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s</span> <span style="color: #9ece6a;">Kahuna</span> <span style="color: #9ece6a;">Short</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Allegra</span> <span style="color: #9ece6a;">K</span> <span style="color: #9ece6a;">Woman</span> <span style="color: #9ece6a;">Faux</span> <span style="color: #9ece6a;">Crystal</span> <span style="color: #9ece6a;">Detail</span> <span style="color: #9ece6a;">Leopard</span> <span style="color: #9ece6a;">Print</span> <span style="color: #9ece6a;">Strapped</span> <span style="color: #9ece6a;">Tank</span> <span style="color: #9ece6a;">Top</span> <span style="color: #9ece6a;">M</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Fox</span> <span style="color: #9ece6a;">Womens</span> <span style="color: #9ece6a;">Juniors</span> <span style="color: #9ece6a;">Trials</span> <span style="color: #9ece6a;">Pullover</span> <span style="color: #9ece6a;">Hoody</span> <span style="color: #9ece6a;">Sweater</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">NULL</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Russell</span> <span style="color: #9ece6a;">Athletic</span> <span style="color: #9ece6a;">Women</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Dri-Power Fleece Mid Rise Pant | NULL |</span></div><div><span style="color: #9ece6a;">| A. Byer Juniors Tropical Cambridge Pant | NULL |</span></div><div><span style="color: #9ece6a;">| Lily Of France Womens Sport In Action Sport Bra | NULL |</span></div><div><span style="color: #9ece6a;">+------------------------------------------------------------------------------------------------------------------------+----------+</span></div><div><span style="color: #9ece6a;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></pre><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p></div><h2 style="text-align: left;"><span style="font-family: Noto Sans;">CROSS JOIN</span></h2><div><span style="font-family: Noto Sans; font-size: medium;">Cross join returns Cartesian product of joining tables, each row from table A join with each row from table B, so the result set of table A with M rows and table B with N rows will be product of rows from table A and table B (MxN rows).</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjsB_G3NSN_pCna4lhQX0B8NqIXRwCAFQzhBG9i3oObIPDroA_GzA4jXt4GMkEVMpNzXsOb8HR8tMMV1f-3UG5FjHAolIrMYfEAmJ4nM40lg1D9xJON9mM0WX9A6UQ1nlU5eE7CIG9YQBh1raA_b07GT2kOmq_XG3s0pPhynBsJ3HOAgtmYjp6gperK7Q" style="margin-left: auto; margin-right: auto;"><img alt="BigQuery - SQL Cross Join" data-original-height="1182" data-original-width="1866" src="https://blogger.googleusercontent.com/img/a/AVvXsEjsB_G3NSN_pCna4lhQX0B8NqIXRwCAFQzhBG9i3oObIPDroA_GzA4jXt4GMkEVMpNzXsOb8HR8tMMV1f-3UG5FjHAolIrMYfEAmJ4nM40lg1D9xJON9mM0WX9A6UQ1nlU5eE7CIG9YQBh1raA_b07GT2kOmq_XG3s0pPhynBsJ3HOAgtmYjp6gperK7Q=s16000" title="BigQuery - SQL Cross Join" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">BigQuery - SQL Cross Join</td></tr></tbody></table><br /></span></div><h3 style="text-align: left;"><span style="font-family: "Noto Sans"; font-size: large;">Example:</span></h3><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">List all possible products a customer can buy</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><div style="line-height: 20px;"><div> <span style="color: #7dcfff;">SELECT</span> </div><div> <span style="color: #0db9d7;">CONCAT</span>(<span style="color: #e0af68;">u</span>.<span style="color: #e0af68;">first_name</span>, <span style="color: #89ddff;">"</span><span style="color: #9ece6a;"> </span><span style="color: #89ddff;">"</span>, <span style="color: #e0af68;">u</span>.<span style="color: #e0af68;">last_name</span>) <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">Customer Name</span><span style="color: #89ddff;">`</span>,</div><div> <span style="color: #e0af68;">p</span>.<span style="color: #e0af68;">name</span> <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">Product Name</span><span style="color: #89ddff;">`</span></div><div> <span style="color: #7dcfff;">FROM</span></div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.users</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> u</div><div> <span style="color: #89ddff;">CROSS</span> <span style="color: #7dcfff;">JOIN</span> </div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.products</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> p</div><div> <span style="color: #7dcfff;">LIMIT</span> <span style="color: #ff9e64;">10</span>;</div></div></div></div><div><div style="line-height: 20px;"><div style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><div style="line-height: 20px;"><div style="color: #a9b1d6;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><br /></div><div style="line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #c0caf5;">+---------------+---------------------------------------------------------------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Customer</span> <span style="color: #9ece6a;">Name</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Product</span> <span style="color: #9ece6a;">Name</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+---------------+---------------------------------------------------------------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Michelle</span> <span style="color: #9ece6a;">Rowe</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">TYR</span> <span style="color: #9ece6a;">Alliance</span> <span style="color: #9ece6a;">Team</span> <span style="color: #9ece6a;">Splice</span> <span style="color: #9ece6a;">Jammer</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Michelle</span> <span style="color: #9ece6a;">Rowe</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">TYR</span> <span style="color: #9ece6a;">Sport</span> <span style="color: #9ece6a;">Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Solid Racer Swim Suit |</span></div><div><span style="color: #9ece6a;">| Michelle Rowe | TYR Sport Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s</span> <span style="color: #9ece6a;">Square</span> <span style="color: #9ece6a;">Leg</span> <span style="color: #9ece6a;">Short</span> <span style="color: #9ece6a;">Swim</span> <span style="color: #9ece6a;">Suit</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Michelle</span> <span style="color: #9ece6a;">Rowe</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">TYR</span> <span style="color: #9ece6a;">Sport</span> <span style="color: #9ece6a;">Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s 4-Inch Nylon Trainer-A Swim Suit |</span></div><div><span style="color: #9ece6a;">| Michelle Rowe | TYR Sport Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s</span> <span style="color: #9ece6a;">Swim</span> <span style="color: #9ece6a;">Short/Resistance</span> <span style="color: #9ece6a;">Short</span> <span style="color: #9ece6a;">Swim</span> <span style="color: #9ece6a;">Suit</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Michelle</span> <span style="color: #9ece6a;">Rowe</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">TYR</span> <span style="color: #9ece6a;">Sport</span> <span style="color: #9ece6a;">Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Poly Mesh Trainer Swim Suit |</span></div><div><span style="color: #9ece6a;">| Michelle Rowe | 2XU Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s</span> <span style="color: #9ece6a;">Swimmers</span> <span style="color: #9ece6a;">Compression</span> <span style="color: #9ece6a;">Long</span> <span style="color: #9ece6a;">Sleeve</span> <span style="color: #9ece6a;">Top</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Michelle</span> <span style="color: #9ece6a;">Rowe</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">TYR</span> <span style="color: #9ece6a;">Sport</span> <span style="color: #9ece6a;">Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Solid Durafast Jammer Swim Suit |</span></div><div><span style="color: #9ece6a;">| Michelle Rowe | TYR Sport Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s</span> <span style="color: #9ece6a;">Alliance</span> <span style="color: #9ece6a;">Durafast</span> <span style="color: #9ece6a;">Splice</span> <span style="color: #9ece6a;">Square</span> <span style="color: #9ece6a;">Leg</span> <span style="color: #9ece6a;">Swim</span> <span style="color: #9ece6a;">Suit</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Michelle</span> <span style="color: #9ece6a;">Rowe</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">TYR</span> <span style="color: #9ece6a;">Sport</span> <span style="color: #9ece6a;">Men</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Solid Jammer Swim Suit |</span></div><div><span style="color: #9ece6a;">+---------------+---------------------------------------------------------------+</span></div></div></div></div><div> </div></div></div></div></div></div></div></div></div></div></div></pre></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">Implicitly writing cross join using a comma is known as Comma cross join. The below example shows the comma cross join syntax for the same use case.</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div> <span style="color: #7dcfff;">SELECT</span> </div><div> <span style="color: #0db9d7;">CONCAT</span>(<span style="color: #e0af68;">u</span>.<span style="color: #e0af68;">first_name</span>, <span style="color: #89ddff;">"</span><span style="color: #9ece6a;"> </span><span style="color: #89ddff;">"</span>, <span style="color: #e0af68;">u</span>.<span style="color: #e0af68;">last_name</span>) <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">Customer Name</span><span style="color: #89ddff;">`</span>,</div><div> <span style="color: #e0af68;">p</span>.<span style="color: #e0af68;">name</span> <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">Product Name</span><span style="color: #89ddff;">`</span></div><div> <span style="color: #7dcfff;">FROM</span></div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.users</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> u, <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.products</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> p</div><div> <span style="color: #7dcfff;">ORDER BY</span> </div><div> <span style="color: #ff9e64;">1</span> <span style="color: #89ddff;">DESC</span>,</div><div> <span style="color: #ff9e64;">2</span> <span style="color: #89ddff;">DESC</span></div><div> <span style="color: #7dcfff;">LIMIT</span> <span style="color: #ff9e64;">10</span>;</div></div></div></div></div><div><div style="line-height: 20px;"><div style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><div style="line-height: 20px;"><div style="color: #a9b1d6;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><br /></div><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #c0caf5;">+---------------+------------------------------------------------------------------------------------------------------------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Customer</span> <span style="color: #9ece6a;">Name</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Product</span> <span style="color: #9ece6a;">Name</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+---------------+------------------------------------------------------------------------------------------------------------+</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Zoe</span> <span style="color: #9ece6a;">Wood</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Â</span> <span style="color: #9ece6a;">Â Exclusive</span> <span style="color: #9ece6a;">Hawaiian</span> <span style="color: #9ece6a;">Tropic</span> <span style="color: #9ece6a;">Sunset</span> <span style="color: #9ece6a;">In</span> <span style="color: #9ece6a;">Paradise</span> <span style="color: #9ece6a;">Aloha</span> <span style="color: #9ece6a;">Shirt</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Zoe</span> <span style="color: #9ece6a;">Wood</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Â</span> <span style="color: #9ece6a;">Â Exclusive</span> <span style="color: #9ece6a;">Hawaiian</span> <span style="color: #9ece6a;">Sunset</span> <span style="color: #9ece6a;">In</span> <span style="color: #9ece6a;">Paradise</span> <span style="color: #9ece6a;">Aloha</span> <span style="color: #9ece6a;">Shirt</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Zoe</span> <span style="color: #9ece6a;">Wood</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Â</span> <span style="color: #9ece6a;">Â Exclusive</span> <span style="color: #9ece6a;">Hawaiian</span> <span style="color: #9ece6a;">Orchid</span> <span style="color: #89ddff;">(</span><span style="color: #c0caf5;">100%</span> <span style="color: #9ece6a;">cotton</span><span style="color: #89ddff;">)</span> <span style="color: #9ece6a;">Aloha</span> <span style="color: #9ece6a;">Shirt</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Zoe</span> <span style="color: #9ece6a;">Wood</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Â</span> <span style="color: #9ece6a;">Â Exclusive</span> <span style="color: #9ece6a;">Hawaiian</span> <span style="color: #9ece6a;">Flowers</span> <span style="color: #9ece6a;">In</span> <span style="color: #9ece6a;">Paradise</span> <span style="color: #9ece6a;">Aloha</span> <span style="color: #9ece6a;">Shirt</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Zoe</span> <span style="color: #9ece6a;">Wood</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Â</span> <span style="color: #9ece6a;">Â Exclusive</span> <span style="color: #9ece6a;">Hawaiian</span> <span style="color: #9ece6a;">All</span> <span style="color: #9ece6a;">New</span> <span style="color: #9ece6a;">Hibiscus</span> <span style="color: #9ece6a;">In</span> <span style="color: #9ece6a;">Paradise</span> <span style="color: #9ece6a;">Aloha</span> <span style="color: #9ece6a;">Shirt</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Zoe</span> <span style="color: #9ece6a;">Wood</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">white</span> <span style="color: #9ece6a;">Half</span> <span style="color: #9ece6a;">slip</span> <span style="color: #9ece6a;">Culotte</span> <span style="color: #9ece6a;">Underworks</span> <span style="color: #9ece6a;">Pettipants</span> <span style="color: #9ece6a;">nylon</span> <span style="color: #9ece6a;">for</span> <span style="color: #9ece6a;">women</span> <span style="color: #9ece6a;">plus</span> <span style="color: #9ece6a;">size</span> <span style="color: #9ece6a;">trim</span> <span style="color: #9ece6a;">double</span> <span style="color: #9ece6a;">lace</span> <span style="color: #9ece6a;">knickers</span> <span style="color: #9ece6a;">Lingerie</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Zoe</span> <span style="color: #9ece6a;">Wood</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">under.me</span> <span style="color: #9ece6a;">The</span> <span style="color: #9ece6a;">Leggings</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Zoe</span> <span style="color: #9ece6a;">Wood</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">tokidoki</span> <span style="color: #9ece6a;">X</span> <span style="color: #9ece6a;">Marvel</span> <span style="color: #9ece6a;">Women</span> <span style="color: #9ece6a;">Retro</span> <span style="color: #9ece6a;">Spidey</span> <span style="color: #9ece6a;">Dark</span> <span style="color: #9ece6a;">Red</span> <span style="color: #9ece6a;">Hoody</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Zoe</span> <span style="color: #9ece6a;">Wood</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">tokidoki</span> <span style="color: #9ece6a;">X</span> <span style="color: #9ece6a;">Marvel</span> <span style="color: #9ece6a;">Women</span> <span style="color: #9ece6a;">Captain</span> <span style="color: #9ece6a;">Kitty</span> <span style="color: #9ece6a;">Hoodie</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">Zoe</span> <span style="color: #9ece6a;">Wood</span> <span style="color: #89ddff;">|</span> <span style="color: #c0caf5;">tokidoki</span> <span style="color: #9ece6a;">The</span> <span style="color: #9ece6a;">Marvel</span> <span style="color: #9ece6a;">Pop</span> <span style="color: #9ece6a;">Zip</span> <span style="color: #9ece6a;">Hoody</span> <span style="color: #9ece6a;">in</span> <span style="color: #9ece6a;">Dark</span> <span style="color: #9ece6a;">Gray</span> <span style="color: #9ece6a;">Heather</span> <span style="color: #89ddff;">|</span></div><div><span style="color: #c0caf5;">+---------------+------------------------------------------------------------------------------------------------------------+</span></div></div></div></div></div><div> </div><div><br /></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><h1 style="text-align: left;"><span style="font-family: Noto Sans;">BigQuery Joining Strategies</span></h1><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">BigQuery commonly use two joining types: Broadcast join and Hashing join. Typically, when one of the tables is smaller (in few hundred MBs) BigQuery uses Broadcast join and when two large tables BigQuery uses Hashing join, let's discuss about these join types.</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Broadcast Join</span></h2><div><br /></div><div><span style="font-family: Noto Sans; font-size: medium;">BigQuery uses broadcasting join when one of the table is smaller (few hundred MBs in size). Logically, when performing join operation in a distributed setup, to join a row from left side table to rows from right side table, we need to bring together both the rows into single worker node. When joining, if one of the table is smaller, it is efficient to broadcast/send rows from that table to worker where rows from larger table is compared.</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">Typically, when joining, key columns are used to match the rows from both the tables, let say the key value from left side smaller table is 27 matched with every rows from right side larger table with key 27, the broadcast join will send this key (rows from smaller table) to every worker where rows from larger table is computed. Therefore, by broadcasting BigQuery ensures rows from both the sides are colocated and performs local join efficiently.</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">Let's understand broadcast join with an example. We have a small table distribution center table with 10 rows and a large table inventory items with 491237 rows. In terms of table size it is not that big, only 53 MBs, but for understanding the broadcast join, this is suffice. Table description given below:</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">Distribution Centers:</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br /></span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div>❯ bq show <span style="color: #444b6a; font-style: italic;">--project_id=bigquery-public-data thelook_ecommerce.distribution_centers</span></div><div><span style="color: #89ddff;"><br /></span></div><div><span style="color: #89ddff;">Table</span> bigquery<span style="color: #89ddff;">-</span>public<span style="color: #89ddff;">-data</span>:<span style="color: #e0af68;">thelook_ecommerce</span>.<span style="color: #e0af68;">distribution_centers</span></div><br /><div> <span style="color: #89ddff;">Last</span> modified Schema Total <span style="color: #89ddff;">Rows</span> Total Bytes Expiration <span style="color: #89ddff;">Time</span> Partitioning Clustered Fields Total Logical Bytes Total Physical Bytes Labels</div><div><span style="color: #89ddff;"> </span><span style="color: #444b6a; font-style: italic;">----------------- --------------------- ------------ ------------- ------------ ------------------- ------------------ --------------------- ---------------------- --------</span></div><div> <span style="color: #ff9e64;">23</span> Mar <span style="color: #ff9e64;">10</span>:<span style="color: #ff9e64;">06</span>:<span style="color: #ff9e64;">38</span> |<span style="color: #89ddff;">-</span> id: <span style="color: #89ddff;">integer</span> <span style="color: #ff9e64;">10</span> <span style="color: #ff9e64;">409</span> <span style="color: #ff9e64;">409</span> <span style="color: #ff9e64;">13988</span></div><div> |<span style="color: #89ddff;">-</span> <span style="color: #89ddff;">name</span>: string</div><div> |<span style="color: #89ddff;">-</span> latitude: <span style="color: #89ddff;">float</span></div><div> |<span style="color: #89ddff;">-</span> longitude: <span style="color: #89ddff;">float</span></div></div><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><br /></div></div></div></div></div></div></pre></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;">Inventory Items:</span></p><div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br /></span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><div>❯ bq show <span style="color: #444b6a; font-style: italic;">--project_id=bigquery-public-data thelook_ecommerce.inventory_items</span></div><div><span style="color: #89ddff;"><br /></span></div><div><span style="color: #89ddff;">Table</span> bigquery<span style="color: #89ddff;">-</span>public<span style="color: #89ddff;">-data</span>:<span style="color: #e0af68;">thelook_ecommerce</span>.<span style="color: #e0af68;">inventory_items</span></div><div style="line-height: 20px;"><br /></div><div style="line-height: 20px;"><div> <span style="color: #89ddff;">Last</span> modified Schema Total <span style="color: #89ddff;">Rows</span> Total Bytes Total Logical Bytes Total Physical Bytes Labels</div><div><span style="color: #89ddff;"> </span><span style="color: #444b6a; font-style: italic;">----------------- -------------------------------------------- ------------ ------------- --------------------- ---------------------- --------</span></div><div> <span style="color: #ff9e64;">23</span> Mar <span style="color: #ff9e64;">10</span>:<span style="color: #ff9e64;">06</span>:<span style="color: #ff9e64;">47</span> |<span style="color: #89ddff;">-</span> id: <span style="color: #89ddff;">integer</span> <span style="color: #ff9e64;">491237</span> <span style="color: #ff9e64;">81455014</span> <span style="color: #ff9e64;">81455014</span> <span style="color: #ff9e64;">53738166</span></div><div> |<span style="color: #89ddff;">-</span> product_id: <span style="color: #89ddff;">integer</span></div><div> |<span style="color: #89ddff;">-</span> created_at: <span style="color: #89ddff;">timestamp</span></div><div> |<span style="color: #89ddff;">-</span> sold_at: <span style="color: #89ddff;">timestamp</span></div><div> |<span style="color: #89ddff;">-</span> cost: <span style="color: #89ddff;">float</span></div><div> |<span style="color: #89ddff;">-</span> product_category: string</div><div> |<span style="color: #89ddff;">-</span> product_name: string</div><div> |<span style="color: #89ddff;">-</span> product_brand: string</div><div> |<span style="color: #89ddff;">-</span> product_retail_price: <span style="color: #89ddff;">float</span></div><div> |<span style="color: #89ddff;">-</span> product_department: string</div><div> |<span style="color: #89ddff;">-</span> product_sku: string</div><div> |<span style="color: #89ddff;">-</span> product_distribution_center_id: <span style="color: #89ddff;">integer</span></div></div><div> </div></div></div></span></div></div></div></pre></div><div><br /></div></div><div><br /></div><div><span style="font-family: Noto Sans; font-size: medium;">Join tables to get Product name and distribution center name:</span></div><div><br /></div><div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br /></span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><div><div style="line-height: 20px;"><div> <span style="color: #7dcfff;">SELECT</span></div><div> <span style="color: #e0af68;">iv</span>.<span style="color: #e0af68;">product_name</span> <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">Product Name</span><span style="color: #89ddff;">`</span>,</div><div> <span style="color: #e0af68;">dc</span>.<span style="color: #e0af68;">name</span> <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">Distribution Center</span><span style="color: #89ddff;">`</span></div><div> <span style="color: #7dcfff;">FROM</span></div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.inventory_items</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> iv</div><div> <span style="color: #7dcfff;">INNER JOIN</span></div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.distribution_centers</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> dc</div><div> <span style="color: #89ddff;">ON</span></div><div> <span style="color: #e0af68;">iv</span>.<span style="color: #e0af68;">product_distribution_center_id</span> <span style="color: #bb9af7;">=</span> <span style="color: #e0af68;">dc</span>.<span style="color: #e0af68;">id</span>;</div><div><br /></div><div><br /></div><div><div style="line-height: 20px;"><div><span style="color: #89ddff;">+</span><span style="color: #444b6a; font-style: italic;">----------------------------------------------------------------------------+---------------------+</span></div><div>| Product <span style="color: #89ddff;">Name</span> | <span style="color: #89ddff;">Distribution</span> Center |</div><div><span style="color: #89ddff;">+</span><span style="color: #444b6a; font-style: italic;">----------------------------------------------------------------------------+---------------------+</span></div><div>| Quiksilver Waterman Men<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s On The Rise | Houston TX |</span></div><div><span style="color: #9ece6a;">| Quiksilver Waterman Men</span><span style="color: #89ddff;">'</span>s <span style="color: #89ddff;">On</span> The Rise | Houston TX |</div><div>| Quiksilver Waterman Men<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s On The Rise | Houston TX |</span></div><div><span style="color: #9ece6a;">| Quiksilver Waterman Men</span><span style="color: #89ddff;">'</span>s <span style="color: #89ddff;">On</span> The Rise | Houston TX |</div><div>| KEEN Women Bellingham Low Ultralite Sock | Houston TX |</div><div>| KEEN Women Bellingham Low Ultralite Sock | Houston TX |</div><div>| KEEN Women Bellingham Low Ultralite Sock | Houston TX |</div><div>| Husky Animal Hat <span style="color: #89ddff;">with</span> Mittens | Houston TX |</div><div>| Husky Animal Hat <span style="color: #89ddff;">with</span> Mittens | Houston TX |</div><div>| Husky Animal Hat <span style="color: #89ddff;">with</span> Mittens | Houston TX |</div><div>| Husky Animal Hat <span style="color: #89ddff;">with</span> Mittens | Houston TX |</div><div>| Volcom Men<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Argyle Socks | Houston TX |</span></div><div><span style="color: #9ece6a;">| Volcom Men</span><span style="color: #89ddff;">'</span>s Argyle Socks | Houston TX |</div><div>| Volcom Men<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Argyle Socks | Houston TX |</span></div><div><span style="color: #9ece6a;">| Volcom Men</span><span style="color: #89ddff;">'</span>s Argyle Socks | Houston TX |</div><div>| Volcom Men<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Argyle Socks | Houston TX |</span></div><div><span style="color: #9ece6a;">| Volcom Men</span><span style="color: #89ddff;">'</span>s Argyle Socks | Houston TX |</div><div>| Wendy Glez Rose Cami Black | Houston TX |</div><div>| Wendy Glez Rose Cami Black | Houston TX |</div><div>| Wendy Glez Rose Cami Black | Houston TX |</div><div>| Carhartt Women<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Hooded Knit Jacket | Houston TX |</span></div><div><span style="color: #9ece6a;">| Carhartt Women</span><span style="color: #89ddff;">'</span>s Hooded Knit Jacket | Houston TX |</div><div>| Carhartt Women<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Hooded Knit Jacket | Houston TX |</span></div><div><span style="color: #9ece6a;">| Carhartt Women</span><span style="color: #89ddff;">'</span>s Hooded Knit Jacket | Houston TX |</div><div>| Carhartt Women<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Hooded Knit Jacket | Houston TX |</span></div><div><span style="color: #9ece6a;">| NEW Aluminum Credit Card Wallet - RFID Blocking Case - Pink (New Products) | Houston TX |</span></div><div><span style="color: #9ece6a;">| NEW Aluminum Credit Card Wallet - RFID Blocking Case - Pink (New Products) | Houston TX |</span></div><div><span style="color: #9ece6a;">| NEW Aluminum Credit Card Wallet - RFID Blocking Case - Pink (New Products) | Houston TX |</span></div><div><span style="color: #9ece6a;">+----------------------------------------------------------------------------+---------------------+</span></div></div></div><div><br /></div><div><br /></div><div><br /></div></div></div></div></div></span></div></div></div></pre></div><div><br /></div></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">Let's check the query execution plan:</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjWgfxuIV8QLoB9xvM78VkWv7gQOyMgl0OdSodFHiL5VXGtRKe-Ap-laaN_tCzpAUI2XDSmGRNPlshnZLj50p0giFw_T6rvA8uLpH_R4uANu-W1tMPdrWB9jZJ13y5TNQpFIWdZtO8FNNeaSCJmSQVN4eX109kkR1rfJ4hOR8YXWCOZS6E19xLB98Alnw" style="margin-left: auto; margin-right: auto;"><img alt="BigQuery - Broadcast Join - Execution Plan" data-original-height="1844" data-original-width="3572" src="https://blogger.googleusercontent.com/img/a/AVvXsEjWgfxuIV8QLoB9xvM78VkWv7gQOyMgl0OdSodFHiL5VXGtRKe-Ap-laaN_tCzpAUI2XDSmGRNPlshnZLj50p0giFw_T6rvA8uLpH_R4uANu-W1tMPdrWB9jZJ13y5TNQpFIWdZtO8FNNeaSCJmSQVN4eX109kkR1rfJ4hOR8YXWCOZS6E19xLB98Alnw=s16000" title="BigQuery - Broadcast Join - Execution Plan" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">BigQuery - Broadcast Join</td></tr></tbody></table><br /><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">As you can see from the execution graph there are 3 stages input, joining and output. In S01 stage total 10 rows of id and name columns read operation from distribution centers table, it is described in the below execution plan:</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEj5ZWbV1q6Ml6SrvkUCSj8RTzrFL8wiL7pA5VrHUhI87S8qXigebfZOjjqVNnq_RgE1FP0--fXhyUluuPsXoc0PiMXK3w8S7MQ7lCEsdxmmAoanWg0KKrdRcDODP_EClH01VKBWQ8yDCBYP61xf7Renb3--bszdZowC3TpxK2UY_Ux8owTI_LPHnqKzSA" style="margin-left: auto; margin-right: auto;"><img alt="BigQuery - Broadcast join - input stage" data-original-height="450" data-original-width="2148" src="https://blogger.googleusercontent.com/img/a/AVvXsEj5ZWbV1q6Ml6SrvkUCSj8RTzrFL8wiL7pA5VrHUhI87S8qXigebfZOjjqVNnq_RgE1FP0--fXhyUluuPsXoc0PiMXK3w8S7MQ7lCEsdxmmAoanWg0KKrdRcDODP_EClH01VKBWQ8yDCBYP61xf7Renb3--bszdZowC3TpxK2UY_Ux8owTI_LPHnqKzSA=s16000" title="BigQuery - Broadcast join - input stage" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">BigQuery - Broadcast join - input stage</td></tr></tbody></table><br /></span><p><span style="font-family: Noto Sans; font-size: medium;">In S02 stage, two READ operations, 10 rows from S01 and 4,91, 247 rows from inventory items table and BigQuery broadcasted all 10 rows to ALL the rows from the right side of the table for matching. This is specified by the keyword </span><span color="rgba(0, 0, 0, 0.66)" style="background-color: white; font-family: Roboto; font-size: 13px;">INNER HASH JOIN EACH WITH ALL </span><span style="font-family: Noto Sans; font-size: medium;">and output is written to S03 with columns Product Name and Distribution Center.</span></p></div><div><span style="font-family: "Noto Sans"; font-size: large;"><br /></span></div><div><span style="font-family: "Noto Sans"; font-size: large;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgbvNrvRwnoOcwKrkwFFn0XoesI5grzFwEVE02maaNFzrcshgfo1SAoYexUQpreoBV9lsEriK9Mbp6Jsi_5LYC31tREcz774onIrPIKoMMJbPI9l1BG_dOcy61tPHgreOalWBoSsxdWzbFIItdsaHhJ0R3M2thTaOJJzDp_aqcZi_iJA_8tC23N4nlPCA" style="margin-left: auto; margin-right: auto;"><img alt="BigQuery - Broadcast Join - Joining Stage" data-original-height="566" data-original-width="2198" src="https://blogger.googleusercontent.com/img/a/AVvXsEgbvNrvRwnoOcwKrkwFFn0XoesI5grzFwEVE02maaNFzrcshgfo1SAoYexUQpreoBV9lsEriK9Mbp6Jsi_5LYC31tREcz774onIrPIKoMMJbPI9l1BG_dOcy61tPHgreOalWBoSsxdWzbFIItdsaHhJ0R3M2thTaOJJzDp_aqcZi_iJA_8tC23N4nlPCA=s16000" title="BigQuery - Broadcast Join - Joining Stage" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">BigQuery - Broadcast Join - Joining Stage</td></tr></tbody></table><br /></span><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Hash Join</span></h2></div><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;">When joining two large tables BigQuery uses Hash joins. This is computationally more expensive join operation. Broadcasting larger table rows to every worker is inefficient, therefore BigQuery hashes table rows from both side of the join operation, this ensures rows with same key (based on hash value) routed and end up in the same buckets, enabling more efficient local join operation.</span></p><span style="font-family: Noto Sans; font-size: medium;"><br />Let's understand hash join with an example, we will cross join products and users table to illustrate hash join. This output of cross join is all possible combination of user and products.</span><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">Users:</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br /></span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div><div style="line-height: 20px;"><div><span style="color: #c0caf5;">❯</span> <span style="color: #9ece6a;">bq</span> <span style="color: #9ece6a;">show</span> <span style="color: #e0af68;">--project_id=bigquery-public-data</span> <span style="color: #9ece6a;">thelook_ecommerce.users</span><span style="color: #89ddff;">;</span></div><br /><div><span style="color: #c0caf5;">Table</span> <span style="color: #9ece6a;">bigquery-public-data:thelook_ecommerce.users</span></div><br /><div> <span style="color: #c0caf5;">Last</span> <span style="color: #9ece6a;">modified</span> <span style="color: #9ece6a;">Schema</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Rows</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Bytes</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Logical</span> <span style="color: #9ece6a;">Bytes</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Physical</span> <span style="color: #9ece6a;">Bytes</span></div><div> <span style="color: #c0caf5;">-----------------</span> <span style="color: #e0af68;">---------------------------</span> <span style="color: #e0af68;">------------</span> <span style="color: #e0af68;">-------------</span> <span style="color: #e0af68;">---------------------</span> <span style="color: #e0af68;">----------------------</span></div><div> <span style="color: #c0caf5;">24</span> <span style="color: #9ece6a;">Mar</span> <span style="color: #9ece6a;">10:23:</span><span style="color: #ff9e64;">38</span> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">id:</span> <span style="color: #9ece6a;">integer</span> <span style="color: #ff9e64;">100000</span> <span style="color: #ff9e64;">15816853</span> <span style="color: #ff9e64;">15816853</span> <span style="color: #ff9e64;">25734386</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">first_name:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">last_name:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">email:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">age:</span> <span style="color: #9ece6a;">integer</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">gender:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">state:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">street_address:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">postal_code:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">city:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">country:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">latitude:</span> <span style="color: #9ece6a;">float</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">longitude:</span> <span style="color: #9ece6a;">float</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">traffic_source:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">created_at:</span> <span style="color: #9ece6a;">timestamp</span></div></div></div></div></span><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><br /></div></div></div></div></div></div></pre></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">Products:</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6; line-height: 20px;"><div><span style="color: #c0caf5;">❯</span> <span style="color: #9ece6a;">bq</span> <span style="color: #9ece6a;">show</span> <span style="color: #e0af68;">--project_id=bigquery-public-data</span> <span style="color: #9ece6a;">thelook_ecommerce.products</span><span style="color: #89ddff;">;</span></div><br /><br /><div><span style="color: #c0caf5;">Table</span> <span style="color: #9ece6a;">bigquery-public-data:thelook_ecommerce.products</span></div><br /><div> <span style="color: #c0caf5;">Last</span> <span style="color: #9ece6a;">modified</span> <span style="color: #9ece6a;">Schema</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Rows</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Bytes</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Logical</span> <span style="color: #9ece6a;">Bytes</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Physical</span> <span style="color: #9ece6a;">Bytes</span> </div><div> <span style="color: #c0caf5;">-----------------</span> <span style="color: #e0af68;">------------------------------------</span> <span style="color: #e0af68;">------------</span> <span style="color: #e0af68;">-------------</span> <span style="color: #e0af68;">---------------------</span> <span style="color: #e0af68;">----------------------</span> </div><div> <span style="color: #c0caf5;">24</span> <span style="color: #9ece6a;">Mar</span> <span style="color: #9ece6a;">10:23:</span><span style="color: #ff9e64;">37</span> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">id:</span> <span style="color: #9ece6a;">integer</span> <span style="color: #ff9e64;">29120</span> <span style="color: #ff9e64;">4285975</span> <span style="color: #ff9e64;">4285975</span> <span style="color: #ff9e64;">9762698</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">cost:</span> <span style="color: #9ece6a;">float</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">category:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">name:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">brand:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">retail_price:</span> <span style="color: #9ece6a;">float</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">department:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">sku:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">distribution_center_id:</span> <span style="color: #9ece6a;">integer</span></div></div><br /></span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><br /></div></div></div></pre></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div></div><div><span style="font-family: Noto Sans; font-size: medium;">Cross join users and products:</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><br /></span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div><div style="line-height: 20px;"><div> <span style="color: #89ddff;">WITH</span> </div><div> users <span style="color: #89ddff;">as</span> (</div><div> <span style="color: #7dcfff;">SELECT</span></div><div> id,</div><div> <span style="color: #0db9d7;">CONCAT</span>(first_name, <span style="color: #89ddff;">"</span><span style="color: #9ece6a;"> </span><span style="color: #89ddff;">"</span>, last_name) <span style="color: #89ddff;">as</span> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">CustomerName</span><span style="color: #89ddff;">`</span></div><div> <span style="color: #7dcfff;">FROM</span></div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.users</span><span style="color: #89ddff;">`</span>),</div><br /><div> products <span style="color: #89ddff;">as</span> (</div><div> <span style="color: #7dcfff;">SELECT</span></div><div> id,</div><div> <span style="color: #89ddff;">name</span>,</div><div> distribution_center_id</div><div> <span style="color: #7dcfff;">FROM</span></div><div> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.products</span><span style="color: #89ddff;">`</span>)</div><br /><div> <span style="color: #7dcfff;">SELECT</span> <span style="color: #89ddff;">*</span> <span style="color: #7dcfff;">FROM</span> users <span style="color: #89ddff;">CROSS</span> <span style="color: #7dcfff;">JOIN</span> products;</div><div><br /></div><div><br /></div><div><div style="line-height: 20px;"><div><span style="color: #89ddff;">+</span><span style="color: #444b6a; font-style: italic;">-------+------------------+-------+---------------------------------------------------------------+------------------------+</span></div><div>| id | CustomerName | id_1 | <span style="color: #89ddff;">name</span> | distribution_center_id |</div><div><span style="color: #89ddff;">+</span><span style="color: #444b6a; font-style: italic;">-------+------------------+-------+---------------------------------------------------------------+------------------------+</span></div><div>| <span style="color: #ff9e64;">96585</span> | Michael Anderson | <span style="color: #ff9e64;">27457</span> | TYR Sport Men<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Solid Durafast Jammer Swim Suit | 1 |</span></div><div><span style="color: #9ece6a;">| 96585 | Michael Anderson | 27466 | TYR Sport Men</span><span style="color: #89ddff;">'</span>s Swim Short<span style="color: #89ddff;">/</span>Resistance Short Swim Suit | <span style="color: #ff9e64;">1</span> |</div><div>| <span style="color: #ff9e64;">96585</span> | Michael Anderson | <span style="color: #ff9e64;">27529</span> | TYR Sport Men<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Poly Mesh Trainer Swim Suit | 1 |</span></div><div><span style="color: #9ece6a;">| 96585 | Michael Anderson | 27552 | TYR Sport Men</span><span style="color: #89ddff;">'</span>s Solid Racer Swim Suit | <span style="color: #ff9e64;">1</span> |</div><div>| <span style="color: #ff9e64;">96585</span> | Michael Anderson | <span style="color: #ff9e64;">27487</span> | TYR Sport Men<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s 4-Inch Nylon Trainer-A Swim Suit | 1 |</span></div><div><span style="color: #9ece6a;">| 96585 | Michael Anderson | 27481 | TYR Alliance Team Splice Jammer | 1 |</span></div><div><span style="color: #9ece6a;">| 96585 | Michael Anderson | 27537 | TYR Sport Men</span><span style="color: #89ddff;">'</span>s Alliance Durafast Splice <span style="color: #0db9d7;">Square</span> Leg Swim Suit | <span style="color: #ff9e64;">1</span> |</div><div>| <span style="color: #ff9e64;">96585</span> | Michael Anderson | <span style="color: #ff9e64;">27510</span> | TYR Sport Men<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Solid Jammer Swim Suit | 1 |</span></div><div><span style="color: #9ece6a;">| 96585 | Michael Anderson | 27569 | 2XU Men</span><span style="color: #89ddff;">'</span>s Swimmers <span style="color: #89ddff;">Compression</span> Long Sleeve <span style="color: #89ddff;">Top</span> | <span style="color: #ff9e64;">1</span> |</div><div>| <span style="color: #ff9e64;">96585</span> | Michael Anderson | <span style="color: #ff9e64;">27445</span> | TYR Sport Men<span style="color: #89ddff;">'</span><span style="color: #9ece6a;">s Square Leg Short Swim Suit | 1 |</span></div><div><span style="color: #9ece6a;">+-------+------------------+-------+---------------------------------------------------------------+------------------------+</span></div></div></div></div></div><div><br /></div></div></div></div></span></div></div></div></pre></div><div><br style="font-family: Times; font-size: medium;" /></div></span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div></div><div><span style="font-family: Noto Sans; font-size: medium;">Execution plan:</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiNPLdGHp8UGgsdMLVmYfnnCjvYtD5tYVeUhT-OuPlp9W9TYhHZ_XlNXSjtSjYf6cSS7UePy6S479Yjia0FXp0ipFPdZx7wi21I2rQ19sWF9XcANSbV4_bPyIQbBne0LjI5HbxHZStu1LBLYHbyu7hExCoko6gRare_0jiPdPcBrQ9M7Qbu_ZvnvfT0kA" style="margin-left: auto; margin-right: auto;"><img alt="BigQuery - Hash Join - Execution Plan" data-original-height="1850" data-original-width="3580" src="https://blogger.googleusercontent.com/img/a/AVvXsEiNPLdGHp8UGgsdMLVmYfnnCjvYtD5tYVeUhT-OuPlp9W9TYhHZ_XlNXSjtSjYf6cSS7UePy6S479Yjia0FXp0ipFPdZx7wi21I2rQ19sWF9XcANSbV4_bPyIQbBne0LjI5HbxHZStu1LBLYHbyu7hExCoko6gRare_0jiPdPcBrQ9M7Qbu_ZvnvfT0kA=s16000" title="BigQuery - Hash Join - Execution Plan" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">BigQuery - Hash Join - Execution Plan</td></tr></tbody></table><br /></span><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;">We are interested in S03 Join stage, here BigQuery performs </span><span style="font-family: Source Code Pro;">EACH WITH EACH</span><span style="font-family: Noto Sans; font-size: medium;"> matching, i.e every row from left side of table is matched with every row from right side of the table. Additionally, we are limiting the number of rows to 100 to avoid CROSS JOIN output errors.</span></p></div><div><span style="font-family: Noto Sans; font-size: medium;"> </span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhE-bYs2dEoS57xCWaugWNWld5GOqWGM26im4w7ASPArykkYB_ooLpFtN3eW6YMtmKKGHKyLHZNt21EVniHxVGMty8M8otk0fX4gCOmvVma4QxdW1JX_vodMnntHijl997TbGI63ed113cBHUP-uBp60MquodpshcR2KhotSijkSwzits8tpa152BZR1Q" style="margin-left: auto; margin-right: auto;"><img alt="BigQuery - Hash Join - EACH WITH EACH" data-original-height="584" data-original-width="2086" src="https://blogger.googleusercontent.com/img/a/AVvXsEhE-bYs2dEoS57xCWaugWNWld5GOqWGM26im4w7ASPArykkYB_ooLpFtN3eW6YMtmKKGHKyLHZNt21EVniHxVGMty8M8otk0fX4gCOmvVma4QxdW1JX_vodMnntHijl997TbGI63ed113cBHUP-uBp60MquodpshcR2KhotSijkSwzits8tpa152BZR1Q=s16000" title="BigQuery - Hash Join - EACH WITH EACH" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">BigQuery - Hash Join - EACH WITH EACH</td></tr></tbody></table></span><div><br /></div><div><div><br /></div><h1 style="text-align: left;"><span style="font-family: Noto Sans;">How to optimize BigQuery Join operations?</span></h1><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">BigQuery recommends following best practices for improving join performance.</span></div><h2 style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></h2><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Avoiding Self-joins</span></h2><div><span style="font-family: Noto Sans; font-size: medium;">In Self-join, a table is joined with itself. For example, you may self join employee table to get employee and supervisor. As self-joins are typically compute row-dependant relationships, it may potentially squares the number of output rows. So recommended best practices is to use <a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/window-function-calls" rel="nofollow" target="_blank">analytic window function</a>.</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><h2 style="text-align: left;"><span style="font-family: "Noto Sans";">Avoid joins that generate more output than input</span></h2><span style="font-family: Noto Sans;"><span style="font-size: medium;">Typically, when use Cross join two big tables, every row from first table is joined with every row in second table, you may observed the query may not be completing as well. It is recommended to avoid cross joins not just in terms of performance, it impacts the overall cost/slot usage as well.<br /><br />To avoid performance issues associated with joins that generate more output than input, use aggregate functions to pre-aggregate the data or use window functions.</span></span></div><div><span style="font-family: Noto Sans;"><span style="font-size: medium;"><br /></span></span><h2><span style="font-family: Noto Sans;">Join on integer data type columns</span></h2><div><span style="font-family: Noto Sans; font-size: medium;">It is easier to compare the fixed length Integer columns than variable length string columns, so if when possible use integer data type column to join tables.</span></div><div><br /></div><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Broadcast Joins</span></h2><div><span style="font-family: Noto Sans; font-size: medium;">When you join multiple tables with larger number of rows and smaller number of rows, it is recommended to place the table with larger number of rows first, followed by tables with smaller number of rows and then place the other tables in decreasing size.</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">Like other database systems, BigQuery optimizer can determine the best execution plan based on the table statistics, still it is recommended to place the tables with larger rows first.</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Hash Joins</span></h2><div><span style="font-family: Noto Sans; font-size: medium;">In broadcast join (i.e) joining a larger table and smaller table, the system sends smaller table's records to every slots that handles join operation for efficient matching. But when there is no smaller tables, the system cannot send larger table records, so BigQuery uses hash and shuffle operation to shuffle first and second table to fetch the matching keys and sends to a same slot for performing local join. Data shuffling is an expensive operation and it impacts the query performance. Using partitioning and clustering improves overall data shuffling performance. </span></div><div><br /></div><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Reduce the size of data being joined</span></h2><p><span style="font-family: Noto Sans; font-size: medium;">Performing join operation on smaller dataset is better, whenever possible filter the dataset before joining. Rewriting queries to fetch only required data from both the joining tables / subqueries helps in reducing heavy shuffling and other such complex operation on larger datasets and improves overall query performance. When there is a need to perform GROUP BY and JOIN operation on a query, perform aggregation earlier in the query - this will reduce the number of rows joined.</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><h1 style="text-align: left;"><span style="font-family: Noto Sans;">Experimenting with BigQuery Nested and Repeated Structures</span></h1><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;">Generally speaking reading from a flattened table is much faster and efficient than reading from joining tables. Denormalizing schema helps us to achieve flatten table structure where events and metrics available in a same table. In BigQuery we can use repeated and nested fields to denormalize a table.</span></p><span style="font-family: Noto Sans; font-size: medium;"><br />Let's analyse the denormalized BigQuery tables using nested and repeated fields and joining normalized table performance with an example.<br /><br />We have two tables Orders and Orders Nested. In Orders table, we have user_id column which references id column of Users table and orders_nested table is a denormalized table, where users details are stored in a nested column with STRUCT data type.</span><p><span style="font-family: "Noto Sans"; font-size: large;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;">Orders:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6; line-height: 20px;"><span style="color: #c0caf5;">❯</span> <span style="color: #9ece6a;">bq</span> <span style="color: #9ece6a;">show</span> <span style="color: #e0af68;">--project_id=bigquery-public-data</span> <span style="color: #9ece6a;">thelook_ecommerce.orders</span><span style="color: #89ddff;">;</span></div><div style="color: #a9b1d6; line-height: 20px;"><span style="color: #89ddff;"><br /></span></div><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #c0caf5;">Table</span> <span style="color: #9ece6a;">bigquery-public-data:thelook_ecommerce.orders</span></div><br /><div> <span style="color: #c0caf5;">Last</span> <span style="color: #9ece6a;">modified</span> <span style="color: #9ece6a;">Schema</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Rows</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Bytes</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Logical</span> <span style="color: #9ece6a;">Bytes</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Physical</span> <span style="color: #9ece6a;">Bytes</span> </div><div> <span style="color: #c0caf5;">-----------------</span> <span style="color: #e0af68;">----------------------------</span> <span style="color: #e0af68;">------------</span> <span style="color: #e0af68;">-------------</span> <span style="color: #e0af68;">---------------------</span> <span style="color: #e0af68;">----------------------</span></div><div> <span style="color: #c0caf5;">25</span> <span style="color: #9ece6a;">Mar</span> <span style="color: #9ece6a;">09:46:</span><span style="color: #f7768e;">0</span><span style="color: #ff9e64;">2</span> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">order_id:</span> <span style="color: #9ece6a;">integer</span> <span style="color: #ff9e64;">125402</span> <span style="color: #ff9e64;">6776727</span> <span style="color: #ff9e64;">6776727</span> <span style="color: #ff9e64;">14873107</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">user_id:</span> <span style="color: #9ece6a;">integer</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">status:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">gender:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">created_at:</span> <span style="color: #9ece6a;">timestamp</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">returned_at:</span> <span style="color: #9ece6a;">timestamp</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">shipped_at:</span> <span style="color: #9ece6a;">timestamp</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">delivered_at:</span> <span style="color: #9ece6a;">timestamp</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">num_of_item:</span> <span style="color: #9ece6a;">integer</span></div></div></div></span><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><br /></div></div></div></div></div></div></pre></div><p><br /></p><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;">Orders Nested:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;"></span></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #c0caf5;">❯</span> <span style="color: #9ece6a;">bq</span> <span style="color: #9ece6a;">show</span> <span style="color: #9ece6a;">BigQueryLab.orders_nested</span><span style="color: #89ddff;">;</span></div><br /><div><span style="color: #c0caf5;">Table</span> <span style="color: #9ece6a;">bpo-bu-prod-datawarehouse:BigQueryLab.orders_nested</span></div><br /><div> <span style="color: #c0caf5;">Last</span> <span style="color: #9ece6a;">modified</span> <span style="color: #9ece6a;">Schema</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Rows</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Bytes</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Logical</span> <span style="color: #9ece6a;">Bytes</span> <span style="color: #9ece6a;">Total</span> <span style="color: #9ece6a;">Physical</span> <span style="color: #9ece6a;">Bytes</span> </div><div> <span style="color: #c0caf5;">-----------------</span> <span style="color: #e0af68;">----------------------------</span> <span style="color: #e0af68;">------------</span> <span style="color: #e0af68;">-------------</span> <span style="color: #e0af68;">---------------------</span> <span style="color: #e0af68;">----------------------</span></div><div> <span style="color: #c0caf5;">25</span> <span style="color: #9ece6a;">Mar</span> <span style="color: #9ece6a;">13:19:</span><span style="color: #ff9e64;">24</span> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">order_id:</span> <span style="color: #9ece6a;">integer</span> <span style="color: #ff9e64;">125402</span> <span style="color: #ff9e64;">10172371</span> <span style="color: #ff9e64;">10172371</span> <span style="color: #ff9e64;">2812454</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">status:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">gender:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">created_at:</span> <span style="color: #9ece6a;">timestamp</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">returned_at:</span> <span style="color: #9ece6a;">timestamp</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">shipped_at:</span> <span style="color: #9ece6a;">timestamp</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">delivered_at:</span> <span style="color: #9ece6a;">timestamp</span></div><div> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">num_of_item:</span> <span style="color: #9ece6a;">integer</span></div><div> <span style="color: #c0caf5;">+-</span> <span style="color: #9ece6a;">users:</span> <span style="color: #9ece6a;">record</span></div><div> <span style="color: #89ddff;">|</span> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">user_id:</span> <span style="color: #9ece6a;">integer</span></div><div> <span style="color: #89ddff;">|</span> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">first_name:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">last_name:</span> <span style="color: #9ece6a;">string</span></div><div> <span style="color: #89ddff;">|</span> <span style="color: #89ddff;">|</span><span style="color: #c0caf5;">-</span> <span style="color: #9ece6a;">city:</span> <span style="color: #9ece6a;">string</span></div><div><span style="color: #9ece6a;"><br /></span></div></div></div></span></div></div></div></pre></div><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;">We will get all the orders and corresponding user_id, first_name, last_name and city columns using above tables and analyse its execution plan.</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;">Joining Orders and Users table:</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span><span style="color: #a9b1d6;"><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><div style="color: #a9b1d6;"> <span style="color: #7dcfff;">SELECT</span></div><div style="color: #a9b1d6;"> orders.<span style="color: #89ddff;">*</span>,</div><div style="color: #a9b1d6;"> <span style="color: #e0af68;">users</span>.<span style="color: #e0af68;">first_name</span>,</div><div style="color: #a9b1d6;"> <span style="color: #e0af68;">users</span>.<span style="color: #e0af68;">last_name</span>,</div><div style="color: #a9b1d6;"> <span style="color: #e0af68;">users</span>.<span style="color: #e0af68;">city</span></div><div style="color: #a9b1d6;"> <span style="color: #7dcfff;">FROM</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.orders</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> orders</div><div style="color: #a9b1d6;"> <span style="color: #7dcfff;">INNER JOIN</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">`</span><span style="color: #9ece6a;">bigquery-public-data.thelook_ecommerce.users</span><span style="color: #89ddff;">`</span> <span style="color: #89ddff;">as</span> users</div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">ON</span></div><div style="color: #a9b1d6;"> <span style="color: #e0af68;">orders</span>.<span style="color: #e0af68;">user_id</span> <span style="color: #bb9af7;">=</span> <span style="color: #e0af68;">users</span>.<span style="color: #e0af68;">id</span>;</div><span style="color: #a9b1d6;"><br /></span><div style="line-height: 20px;"><div style="color: #a9b1d6;"><div style="line-height: 20px;"><div><span style="color: #89ddff;">+</span><span style="color: #444b6a; font-style: italic;">----------+---------+---------+--------+---------------------+----------------------+------------+-----------+----------------+</span></div><div>| order_id | <span style="color: #0db9d7;">user_id</span> | <span style="color: #89ddff;">status</span> | gender | created_at | shipped_at | first_name | last_name | city |</div><div><span style="color: #89ddff;">+</span><span style="color: #444b6a; font-style: italic;">----------+---------+---------+--------+---------------------+----------------------+------------+-----------+----------------+</span></div><div>| <span style="color: #ff9e64;">15</span> | <span style="color: #ff9e64;">15</span> | Shipped | F | <span style="color: #ff9e64;">2023</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">03</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">24</span> <span style="color: #ff9e64;">13</span>:<span style="color: #ff9e64;">06</span>:<span style="color: #ff9e64;">16</span> | <span style="color: #ff9e64;">2023</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">03</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">24</span> <span style="color: #ff9e64;">18</span>:<span style="color: #ff9e64;">29</span>:<span style="color: #ff9e64;">16</span> | Jennifer | Morris | City of Yantai |</div><div>| <span style="color: #ff9e64;">21</span> | <span style="color: #ff9e64;">20</span> | Shipped | F | <span style="color: #ff9e64;">2021</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">07</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">27</span> <span style="color: #ff9e64;">03</span>:<span style="color: #ff9e64;">41</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2021</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">07</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">28</span> <span style="color: #ff9e64;">00</span>:<span style="color: #ff9e64;">11</span>:<span style="color: #ff9e64;">00</span> | Alice | Davis | Fort Worth |</div><div>| <span style="color: #ff9e64;">22</span> | <span style="color: #ff9e64;">20</span> | Shipped | F | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">08</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">24</span> <span style="color: #ff9e64;">03</span>:<span style="color: #ff9e64;">41</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">08</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">25</span> <span style="color: #ff9e64;">11</span>:<span style="color: #ff9e64;">06</span>:<span style="color: #ff9e64;">00</span> | Alice | Davis | Fort Worth |</div><div>| <span style="color: #ff9e64;">26</span> | <span style="color: #ff9e64;">23</span> | Shipped | F | <span style="color: #ff9e64;">2020</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">10</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">12</span> <span style="color: #ff9e64;">16</span>:<span style="color: #ff9e64;">54</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2020</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">10</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">13</span> <span style="color: #ff9e64;">08</span>:<span style="color: #ff9e64;">14</span>:<span style="color: #ff9e64;">00</span> | Kristy | Miller | Bella Vista |</div><div>| <span style="color: #ff9e64;">27</span> | <span style="color: #ff9e64;">23</span> | Shipped | F | <span style="color: #ff9e64;">2021</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">03</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">19</span> <span style="color: #ff9e64;">16</span>:<span style="color: #ff9e64;">54</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2021</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">03</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">20</span> <span style="color: #ff9e64;">17</span>:<span style="color: #ff9e64;">46</span>:<span style="color: #ff9e64;">00</span> | Kristy | Miller | Bella Vista |</div><div>| <span style="color: #ff9e64;">32</span> | <span style="color: #ff9e64;">27</span> | Shipped | F | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">03</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">06</span> <span style="color: #ff9e64;">09</span>:<span style="color: #ff9e64;">01</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">03</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">07</span> <span style="color: #ff9e64;">10</span>:<span style="color: #ff9e64;">43</span>:<span style="color: #ff9e64;">00</span> | Lindsey | Singh | London |</div><div>| <span style="color: #ff9e64;">51</span> | <span style="color: #ff9e64;">44</span> | Shipped | F | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">05</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">13</span> <span style="color: #ff9e64;">12</span>:<span style="color: #ff9e64;">38</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">05</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">13</span> <span style="color: #ff9e64;">23</span>:<span style="color: #ff9e64;">22</span>:<span style="color: #ff9e64;">00</span> | Jeanne | Lamb | Pontal |</div><div>| <span style="color: #ff9e64;">54</span> | <span style="color: #ff9e64;">46</span> | Shipped | F | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">07</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">06</span> <span style="color: #ff9e64;">14</span>:<span style="color: #ff9e64;">51</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">07</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">08</span> <span style="color: #ff9e64;">07</span>:<span style="color: #ff9e64;">26</span>:<span style="color: #ff9e64;">00</span> | Deborah | Nunez | Chengdu |</div><div>| <span style="color: #ff9e64;">56</span> | <span style="color: #ff9e64;">46</span> | Shipped | F | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">10</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">03</span> <span style="color: #ff9e64;">14</span>:<span style="color: #ff9e64;">51</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">10</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">04</span> <span style="color: #ff9e64;">18</span>:<span style="color: #ff9e64;">25</span>:<span style="color: #ff9e64;">00</span> | Deborah | Nunez | Chengdu |</div><div>| <span style="color: #ff9e64;">57</span> | <span style="color: #ff9e64;">47</span> | Shipped | F | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">04</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">05</span> <span style="color: #ff9e64;">15</span>:<span style="color: #ff9e64;">26</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">04</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">07</span> <span style="color: #ff9e64;">04</span>:<span style="color: #ff9e64;">52</span>:<span style="color: #ff9e64;">00</span> | Maria | Li | Chicago |</div><div><span style="color: #89ddff;">+</span><span style="color: #444b6a; font-style: italic;">----------+---------+---------+--------+---------------------+----------------------+------------+-----------+----------------+</span></div></div></div><div style="color: #a9b1d6;"><br /></div></div></div></span></div></div></div></pre></div><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;">Some of the query statistics given below:</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #c0caf5;"><br /></span></div><div><span style="color: #c0caf5;">"totalBytesProcessed"</span><span style="color: #c0caf5;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">10258525</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">,</span></div><div><span style="color: #c0caf5;">"totalBytesBilled"</span><span style="color: #c0caf5;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">20971520</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">,</span></div><div><span style="color: #c0caf5;">"totalSlotMs"</span><span style="color: #c0caf5;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">2208</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">,</span></div><div><span style="color: #c0caf5;">"finalExecutionDurationMs"</span><span style="color: #c0caf5;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">1431</span><span style="color: #89ddff;">"</span></div><div><span style="color: #89ddff;"><br /></span></div></div></div></span></div></div></div></pre></div><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;">When joining users and orders table, the query processed around 9 MB of data and returned results in 1.4 seconds and used 2208 milliseconds of slot time and total billable bytes is 20MB. Let's get the same result set from running SELECT statement from orders_nested table.</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;">Selecting from orders_nested table:</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6; line-height: 20px;"><div> <span style="color: #7dcfff;">SELECT</span> <span style="color: #89ddff;">*</span> <span style="color: #7dcfff;">FROM</span> <span style="color: #e0af68;">BigQueryLab</span>.<span style="color: #e0af68;">orders_nested</span>;</div><div style="line-height: 20px;"><br /><div style="line-height: 20px;"><div><span style="color: #89ddff;">+</span><span style="color: #444b6a; font-style: italic;">----------+---------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------+</span></div><div>| order_id | <span style="color: #89ddff;">status</span> | gender | created_at | shipped_at | users |</div><div><span style="color: #89ddff;">+</span><span style="color: #444b6a; font-style: italic;">----------+---------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------+</span></div><div>| <span style="color: #ff9e64;">2028</span> | Shipped | F | <span style="color: #ff9e64;">2021</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">05</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">14</span> <span style="color: #ff9e64;">16</span>:<span style="color: #ff9e64;">37</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2021</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">05</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">17</span> <span style="color: #ff9e64;">00</span>:<span style="color: #ff9e64;">45</span>:<span style="color: #ff9e64;">00</span> | {<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">user_id</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">1585</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">first_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Mariah</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">last_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Johnson</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">city</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Xingtai</span><span style="color: #89ddff;">"</span>} |</div><div>| <span style="color: #ff9e64;">2842</span> | Shipped | F | <span style="color: #ff9e64;">2021</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">09</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">07</span> <span style="color: #ff9e64;">01</span>:<span style="color: #ff9e64;">22</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2021</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">09</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">08</span> <span style="color: #ff9e64;">10</span>:<span style="color: #ff9e64;">18</span>:<span style="color: #ff9e64;">00</span> | {<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">user_id</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">2215</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">first_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Jody</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">last_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Smith</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">city</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Detroit</span><span style="color: #89ddff;">"</span>} |</div><div>| <span style="color: #ff9e64;">4134</span> | Shipped | F | <span style="color: #ff9e64;">2020</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">05</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">19</span> <span style="color: #ff9e64;">01</span>:<span style="color: #ff9e64;">02</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2020</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">05</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">20</span> <span style="color: #ff9e64;">22</span>:<span style="color: #ff9e64;">22</span>:<span style="color: #ff9e64;">00</span> | {<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">user_id</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">3247</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">first_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Beth</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">last_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Clark</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">city</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Shanghai</span><span style="color: #89ddff;">"</span>} |</div><div>| <span style="color: #ff9e64;">4817</span> | Shipped | F | <span style="color: #ff9e64;">2020</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">12</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">03</span> <span style="color: #ff9e64;">05</span>:<span style="color: #ff9e64;">35</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2020</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">12</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">06</span> <span style="color: #ff9e64;">00</span>:<span style="color: #ff9e64;">00</span>:<span style="color: #ff9e64;">00</span> | {<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">user_id</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">3799</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">first_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Donna</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">last_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Spencer</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">city</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">South Hill</span><span style="color: #89ddff;">"</span>} |</div><div>| <span style="color: #ff9e64;">5259</span> | Shipped | F | <span style="color: #ff9e64;">2023</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">02</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">20</span> <span style="color: #ff9e64;">00</span>:<span style="color: #ff9e64;">45</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2023</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">02</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">21</span> <span style="color: #ff9e64;">09</span>:<span style="color: #ff9e64;">56</span>:<span style="color: #ff9e64;">00</span> | {<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">user_id</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">4147</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">first_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Kristi</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">last_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Luna</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">city</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Millington</span><span style="color: #89ddff;">"</span>} |</div><div>| <span style="color: #ff9e64;">5279</span> | Shipped | F | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">09</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">08</span> <span style="color: #ff9e64;">00</span>:<span style="color: #ff9e64;">55</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">09</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">08</span> <span style="color: #ff9e64;">06</span>:<span style="color: #ff9e64;">33</span>:<span style="color: #ff9e64;">00</span> | {<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">user_id</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">4160</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">first_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Caroline</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">last_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Gillespie</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">city</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Bloomington</span><span style="color: #89ddff;">"</span>} |</div><div>| <span style="color: #ff9e64;">6412</span> | Shipped | F | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">07</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">29</span> <span style="color: #ff9e64;">02</span>:<span style="color: #ff9e64;">08</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">07</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">31</span> <span style="color: #ff9e64;">02</span>:<span style="color: #ff9e64;">04</span>:<span style="color: #ff9e64;">00</span> | {<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">user_id</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">5065</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">first_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Rachael</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">last_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Wilson</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">city</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Cape Coral</span><span style="color: #89ddff;">"</span>} |</div><div>| <span style="color: #ff9e64;">7139</span> | Shipped | F | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">05</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">12</span> <span style="color: #ff9e64;">13</span>:<span style="color: #ff9e64;">37</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2022</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">05</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">14</span> <span style="color: #ff9e64;">13</span>:<span style="color: #ff9e64;">41</span>:<span style="color: #ff9e64;">00</span> | {<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">user_id</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">5636</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">first_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Kaitlin</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">last_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Christian</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">city</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Houston</span><span style="color: #89ddff;">"</span>} |</div><div>| <span style="color: #ff9e64;">7385</span> | Shipped | F | <span style="color: #ff9e64;">2021</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">02</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">09</span> <span style="color: #ff9e64;">03</span>:<span style="color: #ff9e64;">35</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2021</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">02</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">11</span> <span style="color: #ff9e64;">20</span>:<span style="color: #ff9e64;">08</span>:<span style="color: #ff9e64;">00</span> | {<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">user_id</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">5823</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">first_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Jasmine</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">last_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Stewart</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">city</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Zhangzhou</span><span style="color: #89ddff;">"</span>} |</div><div>| <span style="color: #ff9e64;">7692</span> | Shipped | F | <span style="color: #ff9e64;">2020</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">11</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">19</span> <span style="color: #ff9e64;">02</span>:<span style="color: #ff9e64;">29</span>:<span style="color: #ff9e64;">00</span> | <span style="color: #ff9e64;">2020</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">11</span><span style="color: #89ddff;">-</span><span style="color: #ff9e64;">21</span> <span style="color: #ff9e64;">15</span>:<span style="color: #ff9e64;">42</span>:<span style="color: #ff9e64;">00</span> | {<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">user_id</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">6074</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">first_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Kim</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">last_name</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Carey</span><span style="color: #89ddff;">"</span>,<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">city</span><span style="color: #89ddff;">"</span>:<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Pinheiro</span><span style="color: #89ddff;">"</span>} |</div><div><span style="color: #89ddff;">+</span><span style="color: #444b6a; font-style: italic;">----------+---------+--------+---------------------+---------------------+-----------------------------------------------------------------------------------------+</span></div></div><div><br /></div><div><br /></div></div></div></span></div></div></div></pre></div><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;">Some of the above query statistics given below:</span></p><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6; line-height: 20px;"><div><div style="line-height: 20px;"><div><span style="color: #c0caf5;">"totalBytesProcessed"</span><span style="color: #c0caf5;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">10172371</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">,</span></div><div><span style="color: #c0caf5;">"totalBytesBilled"</span><span style="color: #c0caf5;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">10485760</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">,</span></div><div><span style="color: #c0caf5;">"totalSlotMs"</span><span style="color: #c0caf5;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">1141</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">,</span></div><div><span style="color: #c0caf5;">"finalExecutionDurationMs"</span><span style="color: #c0caf5;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">1229</span><span style="color: #89ddff;">"</span></div></div></div><div><span style="color: #9ece6a;"><br /></span></div></div></span></div></div></div></pre></div><p><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;">When selecting from the nested table directly without any join, the query processed around 9 MB of data and returned results in 1.2 seconds and used 1141 milliseconds of slot time and total billable bytes is 10 MB.</span></p><p><span style="font-family: "Noto Sans"; font-size: large;"><br /></span></p><p><span style="font-family: Noto Sans; font-size: medium;">Following observations made from this experimentation:</span></p><p></p><ul style="text-align: left;"><li><span style="font-family: Noto Sans; font-size: medium;">Read from nested table is comparatively faster than joining from multiple tables.</span></li><li><span style="font-family: Noto Sans; font-size: medium;">In both scenarios bytes processed is same, but billable bytes processed is higher when joining the tables.</span></li><li><span style="font-family: Noto Sans; font-size: medium;">Total slot time consumed is higher when joining tables and comparatively low when reading from a nested table.</span></li></ul><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">Nested tables looks better based on the current experimentations, however please note:</span></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><ul style="text-align: left;"><li><span style="font-family: Noto Sans; font-size: medium;">The tables used in this experiment is smaller in size with few number of rows - so the difference is not big, you can see few milliseconds of difference in execution time, but when testing with bigger tables, this results may vary.</span></li><li><span style="font-family: Noto Sans; font-size: medium;">The query used is very simple and straightforward, data shuffling is may not be a major consideration here, my guess is when joining bigger tables with uneven distribution, data shuffling will be more.</span></li><li><span style="font-family: Noto Sans; font-size: medium;">When using nested columns, the table size is increased as we store the additional columns in the same table, this increases the storage cost. </span></li></ul><div><span style="font-family: Noto Sans; font-size: medium;">In addition to nested columns, we can use partitioning and clustering features to efficiently store and retrieve data in BigQuery. However, these observations are true for this specific instance, you need to test your queries and use case to find what works best for you.</span></div></div><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><h1 style="text-align: left;"><span style="font-family: Noto Sans;">Summary</span></h1><div><span style="font-family: Noto Sans; font-size: medium;"><br /></span></div><div><span style="font-family: Noto Sans; font-size: medium;">We have covered basic join types such as inner join, left outer join, right outer join, full outer join, cross join, etc. and how to perform join operations in BigQuery and discussed examples with BigQuery public datasets. Also, we have covered common joining strategies in BigQuery such as broadcast join and hash join and explained with an example query execution plan. Finally, we have discussed my experimentation with BigQuery nested and repeated structure and analysed the performance of joining and denormalizing table. I hope this article is helpful and informative. Share your thoughts in comments section, thanks for reading!.</span></div><p></p></div></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com0tag:blogger.com,1999:blog-2027709621250502166.post-7671584261438907292023-09-05T22:31:00.009+05:302023-09-05T22:36:35.442+05:30How to Choose a Data Serialization/Encoding Format? A Practical Guide for Engineers<div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYg6h0YLhCy5Z1c3k7MP0yKV7jodnCv7RQRWKyEaZZFlkWL3Tl9bEns1GjN57-0AO_vRYTTgcY-H9JHLuwn14GtI2XbU_NJM7WE0tHx9xYxHGFSsXLNhKMDVcVmsSUxPni0catpB84qqvetpNKCKn7-QkPDsAem73t_3-unYPWt_iXPm_zPyi17jyN8V5x/s5975/Data%20Encode.jpg" style="margin-left: auto; margin-right: auto;"><img alt="Data Encoding & Decoding. Image Source: Unsplash" border="0" data-original-height="4481" data-original-width="5975" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYg6h0YLhCy5Z1c3k7MP0yKV7jodnCv7RQRWKyEaZZFlkWL3Tl9bEns1GjN57-0AO_vRYTTgcY-H9JHLuwn14GtI2XbU_NJM7WE0tHx9xYxHGFSsXLNhKMDVcVmsSUxPni0catpB84qqvetpNKCKn7-QkPDsAem73t_3-unYPWt_iXPm_zPyi17jyN8V5x/s16000/Data%20Encode.jpg" title="Data Encoding & Decoding. Image Source: Unsplash" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Noto Sans;">Data Encoding & Decoding. Image Source: Unsplash </span></td></tr></tbody></table></div><div><span style="font-family: Noto Sans;"><br /></span></div><p style="text-align: left;"><span style="font-family: Noto Sans;">In the world of software, we often work with different types of data like lists, tables, and more. These data structures are designed to be fast and efficient when our computer programs use them. However, sometimes we need to move this data out of our computer's memory, like when we want to save it to a file or send it over the internet. To do this, we have to change the data into a special format made up of 0s and 1s, which is quite different from data structures. This process is what we call encoding or serialization. </span></p><p style="text-align: left;"><span></span></p><a name='more'></a><p></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">In this article, we'll explore the world of encoding and decoding, which is the reverse process of turning that special format back into usable data. We'll also take a look at different ways to do encoding and decoding, as well as important things to think about when choosing the right method for your software projects.</span></p><div><span style="font-family: Noto Sans;"><br /></span><div><h2 style="text-align: left;"><span style="font-family: Noto Sans;">What is data encoding/decoding (serialization/deserialization):</span></h2><p style="text-align: left;"><span style="font-family: Noto Sans;">The process of converting data into a format that can be easily stored, transmitted, and reconstructed at a later point in time is called data encoding/serialization and the reverse is called data decoding/deserialization. Think of it as packaging your data into a standardized format for safe transportation.</span></p><span style="font-family: Noto Sans;"><br /></span><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Why does data encoding matter?</span></h2><p style="text-align: left;"><span style="font-family: Noto Sans;">In a data-driven world, efficient data encoding and serialization are vital. They enable seamless communication between systems, help save storage space, and enhance data security. Without proper encoding, data can become garbled, lost in transit, or vulnerable to security threats.</span></p><span style="font-family: Noto Sans;"><br /></span><h2 style="text-align: left;"><span style="font-family: Noto Sans;">What are the different encoding formats?</span></h2><p style="text-align: left;"><span style="font-family: Noto Sans;">Encoding formats can be broadly classified into two categories, one is language-specific formats and other language-independent standard formats.</span></p><span style="font-family: Noto Sans;"><br /></span><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Language-specific format </span></h3></div><p style="text-align: left;"><span style="font-family: "Noto Sans";">Let's look at some examples of language-specific encoding formats:</span></p><div><span style="font-family: Noto Sans;"><br /></span><h4 style="text-align: left;"><span style="font-family: Noto Sans;">Python - Pickle:</span></h4><p style="text-align: left;"><span style="font-family: Noto Sans;">Pickle is a Python-specific serialization module that allows you to serialize and deserialize Python objects. It's part of Python's standard library and is primarily used for Python-to-Python data exchange.</span></p><span style="font-family: Noto Sans;"><br />Pickle is valuable when you want to save Python objects to disk, transfer them between processes, or communicate with other Python applications. It preserves the object's state, including complex data structures, functions, and custom classes.<br /><br /></span><h4 style="text-align: left;"><span style="font-family: Noto Sans;">Ruby - Marshal:</span></h4><p style="text-align: left;"><span style="font-family: Noto Sans;">Marshal is Ruby's native serialization format, designed for encoding and decoding Ruby objects. It's a standard part of the Ruby programming language.</span></p><span style="font-family: Noto Sans;"><br />Marshal is used in Ruby for similar reasons as Pickle in Python: to store Ruby objects to disk, transfer them between Ruby processes, or communicate within the Ruby ecosystem. It's efficient and preserves the object's state.<br /><br />Language-specific format example - pickle</span></div><div><span style="font-family: Noto Sans;"><b><br /></b></span><div style="-webkit-font-smoothing: antialiased; border: 0px; box-sizing: border-box; font-family: inherit; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 14px; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-align: justify; text-rendering: optimizelegibility; vertical-align: baseline;"><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 13px; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; text-rendering: optimizelegibility; text-wrap: wrap; vertical-align: middle; width: 984.5px; word-break: normal;"><div style="-webkit-font-smoothing: antialiased; background-color: #1a1b26; border: 0px; box-sizing: border-box; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; line-height: 20px; margin: 0px; padding: 0px; text-rendering: optimizelegibility; vertical-align: baseline;"><span style="-webkit-font-smoothing: antialiased; border: 0px; box-sizing: border-box; color: #c0caf5; font-family: inherit; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-rendering: optimizelegibility; vertical-align: baseline;"><div style="color: #a9b1d6; line-height: 20px; text-wrap: nowrap;"><div><span style="color: #7dcfff;">
import</span> pickle</div><br /><div><span style="color: #444b6a; font-style: italic;"># Data to be serialized using pickle</span></div><div>data <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">author</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">rathish</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">url</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">rathishkumar.com</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">post</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">data encoding/decoding guide</span><span style="color: #89ddff;">"</span></div><div><span style="color: #89ddff;">}</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Serialization (writing) with pickle</span></div><div><span style="color: #bb9af7;">with</span> <span style="color: #0db9d7;">open</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">data.pickle</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">wb</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">)</span> <span style="color: #bb9af7;">as</span> pickle_file<span style="color: #89ddff;">:</span></div><div> pickle<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">dump</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">data</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> pickle_file</span><span style="color: #9abdf5;">)</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Deserialization (reading) with pickle</span></div><div><span style="color: #bb9af7;">with</span> <span style="color: #0db9d7;">open</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">data.pickle</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">rb</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">)</span> <span style="color: #bb9af7;">as</span> pickle_file<span style="color: #89ddff;">:</span></div><div> loaded_data <span style="color: #89ddff;">=</span> pickle<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">load</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">pickle_file</span><span style="color: #9abdf5;">)</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># 'loaded_data' now contains the deserialized data</span></div><div><span style="color: #0db9d7;">print</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">loaded_data</span><span style="color: #9abdf5;">)</span></div><br /></div></span></div></pre></div><div style="-webkit-font-smoothing: antialiased; border: 0px; box-sizing: border-box; font-family: inherit; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 14px; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-align: justify; text-rendering: optimizelegibility; vertical-align: baseline;"><br /></div><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Key Characteristics of language-specific encoding formats:</span></h3><p style="text-align: left;"><span style="font-family: Noto Sans;">Language-specific encoding formats are designed to work specifically within a particular programming language's ecosystem. Here are some key characteristics of such formats:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p></div></div><div style="text-align: left;"><p style="text-align: left;"><b style="font-family: "Noto Sans";"></b></p></div><blockquote><div style="text-align: left;"><p style="text-align: left;"><b style="font-family: "Noto Sans";">Language-Centric: </b><span style="font-family: "Noto Sans";">Language-specific formats are designed to work exclusively within one programming language, making them highly efficient but limiting their compatibility with other languages. </span> </p></div></blockquote><blockquote><span style="font-family: Noto Sans;"><b>Efficiency:</b> These formats are optimized for speed and space within their respective languages, making them a fast and compact choice when working with data native to that language. </span> <span style="font-family: "Noto Sans";"> </span> <br /><span style="font-family: Noto Sans;"><b><br /></b></span><div><span style="font-family: Noto Sans;"><b>Full Object Serialization:</b> They can handle complex language-specific data structures, including custom classes and nested objects, ensuring comprehensive serialization capabilities.</span> <br /> <br /><span style="font-family: "Noto Sans";"><b>Binary Format:</b> Language-specific formats often use binary encoding, a machine-friendly representation, to maximize efficiency in terms of storage and processing.</span> <br /> <br /><span style="font-family: Noto Sans;"><b>Limited Compatibility:</b> Due to their language-specific nature, these formats may struggle to interact with data structures from other programming languages.</span> <br /> <br /><span style="font-family: Noto Sans;"><b>Security Concerns:</b> When deserializing data from untrusted sources, language-specific formats can pose security risks, as they may execute arbitrary code during the deserialization process.</span> <br /> <br /><span style="font-family: Noto Sans;"><b>Version Dependency:</b> Compatibility with language-specific formats is often tied to the version of the language being used, potentially leading to compatibility issues when upgrading.</span> </div><div><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Vendor Lock-In: </b><span style="font-family: "Noto Sans";">Using language-specific formats may lock you into a specific platform or ecosystem, limiting your flexibility in cross-language or cross-platform scenarios.</span></div></blockquote><div><span style="font-family: "Noto Sans";"></span><div><div style="text-align: left;"></div></div><div><div style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></div><div><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Standard Encoding formats:</span></h3><p style="text-align: left;"><span style="font-family: Noto Sans;">Standardized encoding formats can be broadly categorized as textual formats and binary formats. Some examples of textual formats are: JSON, CSV, & XML and binary formats are Thrift, Protocol Buffers, Avro & Apache Parquet, etc</span></p></div><div><span style="font-family: Noto Sans;"><br /></span><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Textual Encoding formats:</span></h3><p style="text-align: left;"><span style="font-family: Noto Sans;">Textual encoding formats are used to represent data in a human-readable text format, which makes them suitable for various applications, including configuration files, data interchange, and more. Here are some common types of textual encoding formats:</span></p><span style="font-family: Noto Sans;"><br /><b>JSON (JavaScript Object Notation):</b> Lightweight, human-readable format with key-value pairs, ideal for web APIs and config files.<br /><br /><b>XML (Extensible Markup Language): </b>Versatile, hierarchical format for data storage, common in web services and documents.<br /><br /><b>YAML (YAML Ain't Markup Language):</b> Designed for config files and data serialization, using indentation for structure and readability.<br /><br /><b>CSV (Comma-Separated Values): </b>Simple tabular data format with values separated by commas, often used in spreadsheets and databases.<br /><br />Textual Encoding formats Example - JSON</span></div><div><span style="font-family: Noto Sans;"><b><br /></b></span></div><div><div style="-webkit-font-smoothing: antialiased; border: 0px; box-sizing: border-box; font-family: inherit; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 14px; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-align: justify; text-rendering: optimizelegibility; vertical-align: baseline;"><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 13px; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; text-rendering: optimizelegibility; text-wrap: wrap; vertical-align: middle; width: 984.5px; word-break: normal;"><div style="-webkit-font-smoothing: antialiased; background-color: #1a1b26; border: 0px; box-sizing: border-box; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; line-height: 20px; margin: 0px; padding: 0px; text-rendering: optimizelegibility; vertical-align: baseline;"><span style="-webkit-font-smoothing: antialiased; border: 0px; box-sizing: border-box; color: #c0caf5; font-family: inherit; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-rendering: optimizelegibility; vertical-align: baseline;"><div style="color: #a9b1d6; line-height: 20px; text-wrap: nowrap;"><div><span style="color: #7dcfff;"><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><div><span style="color: #7dcfff;">import</span> json</div><br /><div><span style="color: #444b6a; font-style: italic;"># Data to be encoded in JSON format</span></div><div>data <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">author</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">rathish</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">url</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">rathishkumar.com</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">post</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">data encoding/decoding guide</span><span style="color: #89ddff;">"</span></div><div><span style="color: #89ddff;">}</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Encoding (writing) JSON data to a file</span></div><div><span style="color: #bb9af7;">with</span> <span style="color: #0db9d7;">open</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">data.json</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">w</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">)</span> <span style="color: #bb9af7;">as</span> json_file<span style="color: #89ddff;">:</span></div><div> json<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">dump</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">data</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> json_file</span><span style="color: #9abdf5;">)</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Decoding (reading) JSON data from the file</span></div><div><span style="color: #bb9af7;">with</span> <span style="color: #0db9d7;">open</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">data.json</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">r</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">)</span> <span style="color: #bb9af7;">as</span> json_file<span style="color: #89ddff;">:</span></div><div> loaded_data <span style="color: #89ddff;">=</span> json<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">load</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">json_file</span><span style="color: #9abdf5;">)</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Now, 'loaded_data' contains the decoded JSON data</span></div><div><span style="color: #0db9d7;">print</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">loaded_data</span><span style="color: #9abdf5;">)</span></div><div><span style="color: #9abdf5;"><br /></span></div></div></div></div></span></div></pre></div><div style="-webkit-font-smoothing: antialiased; border: 0px; box-sizing: border-box; font-family: inherit; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 14px; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-align: justify; text-rendering: optimizelegibility; vertical-align: baseline;"><br /></div><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Key characteristics of textual encoding formats:</span></h3><span style="font-family: Noto Sans;"><br /></span></div></div><b style="font-family: "Noto Sans";"></b><b style="font-family: "Noto Sans";"></b><blockquote><b style="font-family: "Noto Sans";">Human-Readable:</b><span style="font-family: "Noto Sans";"> Textual encoding formats use plain text, making data easily understandable without specialized tools, aiding in debugging and data comprehension.</span><div> <br /><b style="font-family: "Noto Sans";">Structured Data:</b><span style="font-family: "Noto Sans";"> These formats organize data hierarchically, allowing for the representation of complex relationships and structured information.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Interoperability:</b><span style="font-family: "Noto Sans";"> Textual formats work across platforms and programming languages, promoting data exchange and integration in diverse environments.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Open Standard:</b><span style="font-family: "Noto Sans";"> Many are based on open standards with documented specifications, ensuring consistent interpretation and facilitating adoption.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Customizability:</b><span style="font-family: "Noto Sans";"> Users can often define their data structures, adapting the format to specific needs and making it flexible.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Versatility:</b><span style="font-family: "Noto Sans";"> Textual encoding formats find use in various applications, from configuration files to data interchange, making them adaptable.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Readability vs. Efficiency: </b><span style="font-family: "Noto Sans";">While human-readable, they can be less space-efficient for large datasets compared to binary formats.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Parsing Overhead:</b><span style="font-family: "Noto Sans";"> Reading/writing data involves parsing, which may introduce a slight performance overhead.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Encoding Required:</b><span style="font-family: "Noto Sans";"> Non-ASCII characters may need encoding for proper representation, especially for multilingual data.</span> <div style="text-align: left;"><b style="font-family: "Noto Sans";"><br /></b></div><div style="text-align: left;"><b style="font-family: "Noto Sans";">Error-Prone: </b><span style="font-family: "Noto Sans";">Reliance on text makes them susceptible to syntax errors, requiring care in data creation and editing.</span></div></div></blockquote><div><div style="text-align: left;"><span style="font-family: "Noto Sans";"></span></div><div><span style="font-family: "Noto Sans";"></span><div style="text-align: left;"><div style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Binary encoding formats:</span></h3><p style="text-align: left;"><span style="font-family: Noto Sans;">The binary encoding format represents information using sequences of binary digits (0s and 1s). Unlike textual formats, which use human-readable characters, binary formats use machine-friendly representations. For a small dataset, the gains are negligible, but once you get into the terabytes, the choice of data format can have a big impact.</span></p><span style="font-family: Noto Sans;"><br /><b>Apache Avro:</b> Avro is a binary data serialization system used for data exchange between systems in a compact and efficient manner. It is schema-based and provides rich data structures. Avro is commonly used in the Hadoop ecosystem and for communication in distributed systems.<br /><br /><b>Parquet:</b> Parquet is a columnar storage format that stores data efficiently in a binary format. It is particularly popular in big data processing frameworks like Apache Hive, Apache Spark, and Apache Impala. Parquet's columnar organization makes it highly optimized for analytical queries.<br /><br /><b>Apache Thrift:</b> Thrift is a binary protocol used for remote procedure calls (RPC) and data serialization. It allows you to define data types and service interfaces using a language-neutral IDL (Interface Definition Language). Thrift is used in various applications for efficient communication between services.<br /><br /><b>Protocol Buffers (protobuf):</b> Protocol Buffers is a language-agnostic binary serialization format developed by Google. It's used to define data structures and generate language-specific code for serialization and deserialization. Protobuf is known for its efficiency, extensibility, and cross-language compatibility.<br /><br /><b>MessagePack: </b>MessagePack is a binary serialization format that aims to be more compact and efficient than JSON. It's suitable for data exchange between systems and languages and is designed to be simple and fast.<br /><br />Binary encoding format example - Avro</span></div><div style="text-align: left;"><span style="font-family: Noto Sans;"><b><br /></b></span></div><div style="text-align: left;"><div style="-webkit-font-smoothing: antialiased; border: 0px; box-sizing: border-box; font-family: inherit; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 14px; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-align: justify; text-rendering: optimizelegibility; vertical-align: baseline;"><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-size: 13px; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; text-rendering: optimizelegibility; text-wrap: wrap; vertical-align: middle; width: 984.5px; word-break: normal;"><div style="-webkit-font-smoothing: antialiased; background-color: #1a1b26; border: 0px; box-sizing: border-box; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; line-height: 20px; margin: 0px; padding: 0px; text-rendering: optimizelegibility; vertical-align: baseline;"><span style="-webkit-font-smoothing: antialiased; border: 0px; box-sizing: border-box; color: #c0caf5; font-family: inherit; font-feature-settings: inherit; font-kerning: inherit; font-optical-sizing: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-variation-settings: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-rendering: optimizelegibility; vertical-align: baseline;"><div style="color: #a9b1d6; line-height: 20px; text-wrap: nowrap;"><div style="line-height: 20px;"><div><div style="line-height: 20px;"><div><span style="color: #7dcfff;"><br /></span></div><div><span style="color: #7dcfff;">import</span> avro<span style="color: #89ddff;">.</span>schema</div><div><span style="color: #7dcfff;">from</span> avro<span style="color: #89ddff;">.</span>datafile <span style="color: #7dcfff;">import</span> DataFileReader<span style="color: #89ddff;">,</span> DataFileWriter</div><div><span style="color: #7dcfff;">from</span> avro<span style="color: #89ddff;">.</span>io <span style="color: #7dcfff;">import</span> DatumReader<span style="color: #89ddff;">,</span> DatumWriter</div><br /><div><span style="color: #444b6a; font-style: italic;"># Define an Avro schema</span></div><div>schema <span style="color: #89ddff;">=</span> avro<span style="color: #89ddff;">.</span>schema<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">Parse</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">'''</span></div><div><span style="color: #9ece6a;"> {</span></div><div><span style="color: #9ece6a;"> "type": "record",</span></div><div><span style="color: #9ece6a;"> "name": "BlogPost",</span></div><div><span style="color: #9ece6a;"> "fields": [</span></div><div><span style="color: #9ece6a;"> {"name": "author", "type": "string"},</span></div><div><span style="color: #9ece6a;"> {"name": "url", "type": "string"},</span></div><div><span style="color: #9ece6a;"> {"name": "post", "type": "string"}</span></div><div><span style="color: #9ece6a;"> ]</span></div><div><span style="color: #9ece6a;"> }</span></div><div><span style="color: #89ddff;">'''</span><span style="color: #9abdf5;">)</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Data to be serialized using Avro</span></div><div>data <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">author</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">rathish</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">url</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">rathishkumar.com</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">post</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">data encoding/decoding guide</span><span style="color: #89ddff;">"</span></div><div><span style="color: #89ddff;">}</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Serialization (writing) with Avro</span></div><div><span style="color: #bb9af7;">with</span> <span style="color: #0db9d7;">open</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">data.avro</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">wb</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">)</span> <span style="color: #bb9af7;">as</span> avro_file<span style="color: #89ddff;">:</span></div><div> writer <span style="color: #89ddff;">=</span> <span style="color: #7aa2f7;">DataFileWriter</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">avro_file</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #7aa2f7;">DatumWriter</span><span style="color: #9abdf5;">()</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> schema</span><span style="color: #9abdf5;">)</span></div><div> writer<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">append</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">data</span><span style="color: #9abdf5;">)</span></div><div> writer<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">close</span><span style="color: #9abdf5;">()</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Deserialization (reading) with Avro</span></div><div><span style="color: #bb9af7;">with</span> <span style="color: #0db9d7;">open</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">data.avro</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">rb</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">)</span> <span style="color: #bb9af7;">as</span> avro_file<span style="color: #89ddff;">:</span></div><div> reader <span style="color: #89ddff;">=</span> <span style="color: #7aa2f7;">DataFileReader</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">avro_file</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #7aa2f7;">DatumReader</span><span style="color: #9abdf5;">())</span></div><div> loaded_data <span style="color: #89ddff;">=</span> <span style="color: #0db9d7;">next</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">reader</span><span style="color: #9abdf5;">)</span></div><div> reader<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">close</span><span style="color: #9abdf5;">()</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># 'loaded_data' now contains the deserialized Avro data</span></div><div><span style="color: #0db9d7;">print</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">loaded_data</span><span style="color: #9abdf5;">)</span></div><div><span style="color: #9abdf5;"><br /></span></div></div></div></div></div></span></div></pre></div><span style="font-family: Noto Sans;"><br /></span><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Key characteristics of Binary encoding</span></h3><div><span style="font-family: Noto Sans;"><br /></span></div></div></div><div style="text-align: left;"><div style="text-align: left;"><span style="font-family: Noto Sans;"><b></b></span></div></div><b style="font-family: "Noto Sans";"></b><b style="font-family: "Noto Sans";"></b></div><blockquote><div><b style="font-family: "Noto Sans";">Efficiency:</b><span style="font-family: "Noto Sans";"> Binary encoding formats are highly space-efficient and offer faster data serialization and deserialization compared to textual formats. They are optimized for compact storage and efficient data processing.</span> </div><div><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Schema-Based:</b><span style="font-family: "Noto Sans";"> Many binary formats, including Avro and Protocol Buffers, are schema-based. This means they require a predefined schema that defines the structure of the data. This schema helps ensure data consistency and facilitates versioning.</span> </div><div><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Cross-Language Compatibility:</b><span style="font-family: "Noto Sans";"> Formats like Thrift and Protocol Buffers are designed to be language-agnostic, allowing data to be serialized in one language and deserialized in another. This is crucial for systems with components written in different programming languages.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Data Typing: </b><span style="font-family: "Noto Sans";">Binary formats often support a wide range of data types, including primitives, complex structures, and custom types. This enables the representation of diverse data structures and objects.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Serialization and Deserialization:</b><span style="font-family: "Noto Sans";"> Binary formats offer efficient serialization (encoding) and deserialization (decoding) processes. This is crucial for data interchange and storage, especially in high-throughput systems.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Compactness:</b><span style="font-family: "Noto Sans";"> Binary formats produce compact representations of data due to their use of binary encoding. This is especially advantageous when dealing with large datasets or when optimizing storage space.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Columnar Storage (e.g., Parquet):</b><span style="font-family: "Noto Sans";"> Some binary formats, like Parquet, are optimized for columnar storage. This means they store data column-wise rather than row-wise, making them highly efficient for analytical queries.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Forward and Backward Compatibility:</b><span style="font-family: "Noto Sans";"> Schema-based binary formats often provide mechanisms for handling forward and backward compatibility of data schemas. This allows for data evolution without breaking existing systems.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Code Generation: </b><span style="font-family: "Noto Sans";">Formats like Protocol Buffers and Thrift generate language-specific code from the schema. This code simplifies the process of serialization and deserialization, reducing the risk of errors.</span> <br /><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Support for Default Values: </b><span style="font-family: "Noto Sans";">Binary formats often support specifying default values for fields, making it possible to handle missing data or backward compatibility gracefully.</span> </div><div><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Streaming Capabilities:</b><span style="font-family: "Noto Sans";"> Binary formats are often designed to support streaming data. This means data can be read or written incrementally, which is beneficial for handling large datasets or real-time data streams.</span></div><div><b style="font-family: "Noto Sans";"><br /></b></div><div><b style="font-family: "Noto Sans";">Data Integrity: </b><span style="font-family: "Noto Sans";">Binary formats are less prone to data corruption because they lack human-readable characters that might be misinterpreted. This enhances data integrity and reliability.</span></div></blockquote><div><span style="font-family: "Noto Sans";"></span><div style="text-align: left;"><p style="text-align: left;"><span style="font-family: Noto Sans;"></span></p></div><div style="text-align: left;"><div style="text-align: left;"><span style="font-family: Noto Sans;"></span></div></div><div><div style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Key considerations for choosing the data encoding format:</span></h2><span style="font-family: Noto Sans;"><br /></span><p style="text-align: left;"><span style="font-family: Noto Sans;">Choosing the right encoding format is a crucial decision in software development, as it can impact data efficiency, interoperability, and the overall performance of your application. Here are key considerations to help you choose the right encoding format:</span></p><span style="font-family: Noto Sans;"><br /><b>Data Structure and Complexity:</b> Consider the complexity of your data structures. Textual formats like JSON and XML are suitable for hierarchical data, while binary formats like Protocol Buffers and Avro handle more complex, nested structures efficiently.<br /><br /><b>Performance and Efficiency:</b> Evaluate the performance requirements of your application. Binary formats often provide faster serialization and deserialization compared to textual formats. If performance is critical, opt for a binary format.<br /><br /><b>Data Size and Bandwidth:</b> Analyze the size of the data you're transmitting or storing. Binary formats are typically more space-efficient, which can reduce bandwidth usage and storage costs, especially for large datasets.<br /><br /><b>Interoperability:</b> Consider the need to exchange data with systems using different programming languages or technologies. Textual formats like JSON and XML are more interoperable, as they are human-readable and have libraries available in various languages.<br /><br /><b>Schema Flexibility:</b> Assess the need for schema flexibility. Textual formats are often schema-less or allow flexible schemas, making them suitable for evolving data structures. Binary formats like Protocol Buffers and Avro require predefined schemas but provide strong data consistency.<br /><br /><b>Compatibility and Versioning:</b> Think about how data schema changes will be handled over time. Some binary formats have built-in support for schema evolution, while textual formats may require custom solutions for backward and forward compatibility.<br /><br /><b>Security Requirements:</b> Consider security concerns. Binary formats can be more secure against tampering, as they are not easily human-readable. Textual formats may require additional security measures, such as encryption, when transmitting sensitive data.<br /><br /><b>Ease of Debugging and Inspection:</b> Think about how easily you can debug and inspect data. Textual formats are more human-readable and can be viewed directly. Binary formats require specialized tools for inspection.<br /><br /><b>Platform and Ecosystem Constraints:</b> Consider the platforms and ecosystems your application interacts with. Some ecosystems may have conventions or standards that favor specific encoding formats.<br /><br /><b>Use Case and Domain:</b> Tailor your choice to the specific use case and domain of your application. For example, JSON is common for web APIs, while Parquet is well-suited for data warehousing and analytics.<br /><br /><b>Development Time and Resources:</b> Assess the available development resources and expertise. Some encoding formats may have steeper learning curves or require more development effort to implement and maintain.<br /><br /><b>Community and Library Support:</b> Check the availability of libraries and tools for working with your chosen encoding format. A vibrant community and extensive library support can simplify development.<br /><br /><b>Long-Term Considerations:</b> Think about the long-term sustainability of your choice. Ensure the encoding format you choose aligns with your project's roadmap and future requirements.<br /><br /></span><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Comparison of data serialization formats</span></h2></div><div style="text-align: left;"><span style="font-family: Noto Sans;">The below table outlines key attributes, from data structure to interoperability, providing valuable insights for choosing the format that best suits your specific needs.</span></div><div style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></div><div style="text-align: left;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbRKNqZB4Q1V7_-AV1Kozk4R6a1WGm1FURJYcQSAoVp8xteGdGr5_D0FG6wyonjPsA5fg4r_QTaL3qEWHgKi9FcjZd8VgF1l5ekI1jfWSyD2e8DiHA42XE9hHvwn4nlCusNyWjZI2IHpzqULsI7T3lUU4jZz5KzjDFHeOWlkeJxDdtSOiz5V1iigsF3q87/s2616/data%20serialization.png" style="margin-left: auto; margin-right: auto;"><img alt="Comparison of data serialization formats" border="0" data-original-height="1374" data-original-width="2616" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbRKNqZB4Q1V7_-AV1Kozk4R6a1WGm1FURJYcQSAoVp8xteGdGr5_D0FG6wyonjPsA5fg4r_QTaL3qEWHgKi9FcjZd8VgF1l5ekI1jfWSyD2e8DiHA42XE9hHvwn4nlCusNyWjZI2IHpzqULsI7T3lUU4jZz5KzjDFHeOWlkeJxDdtSOiz5V1iigsF3q87/s16000/data%20serialization.png" title="Comparison of data serialization formats" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Noto Sans;">Comparison of data serialization formats</span></td></tr></tbody></table></div><div style="text-align: left;"><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Summary</span></h2><p style="text-align: left;"><span style="font-family: Noto Sans;">In this comprehensive exploration of data encoding and decoding (serialization and deserialization), we've uncovered the fundamental concepts and practical applications of these processes. We delved into various encoding formats, ranging from language-specific solutions like Python's Pickle and Ruby's Marshal to standard formats like JSON, XML, YAML, and binary options such as Apache Avro, Parquet, Thrift, Protocol Buffers, and MessagePack. </span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br />Throughout the article, I've highlighted the key characteristics and considerations for choosing the right encoding format, recognizing that this decision plays a pivotal role in data efficiency, interoperability, and overall software performance. Whether you're optimizing for performance, data size, or compatibility, a thoughtful selection of encoding formats ensures that your data is not only safely transported but also efficiently utilized, aligning with your project's unique requirements and long-term goals.</span></p><h2 style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></h2><h2 style="text-align: left;"><span style="font-family: "Noto Sans";">References</span></h2><div><span style="font-family: Noto Sans;">Serialization - <a href="https://en.wikipedia.org/wiki/Serialization">https://en.wikipedia.org/wiki/Serialization</a></span></div><div><span style="font-family: Noto Sans;">Data serialization in Python - <a href="https://docs.python-guide.org/scenarios/serialization/">https://docs.python-guide.org/scenarios/serialization/</a></span></div><div><span style="font-family: Noto Sans;">Designing data-intensive application - <a href="https://dataintensive.net/">https://dataintensive.net/</a> </span></div><div><span style="font-family: Noto Sans;">Data serialization comparison - <a href="https://encyclopedia.pub/entry/history/show/83003">https://encyclopedia.pub/entry/history/show/83003</a></span></div><div><span style="font-family: Noto Sans;">Compariasion of data serialization formats: <a href="https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats">https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats</a></span></div><div><span style="font-family: "Noto Sans";"><br /></span></div></div></div></div></div></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com0tag:blogger.com,1999:blog-2027709621250502166.post-36412589076972196272023-07-10T10:48:00.032+05:302023-08-19T13:45:15.458+05:30Unlock Advanced Data Visualization: The Complete Guide to Installing and Using Apache Superset on Linux<p><span style="font-family: Noto Sans;">
</span></p><p style="text-align: justify;"></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGk5afX58yS8Nf8YceDYmQShsokfr12Q64kjGJDslA4WRUf-1ltlGFx02MXmFvzwpq6A6Yo2xEtedH2zOADDRHtGV9uJqG1w5O1AzOMBYAdb0v8Fxdq-wd0M2MvUFqhDR2UmeRUvWPZ09WcwnTTc9Roh4YsRQ3oY0vXV8hkRcloVMdNn_eSwNd2W8gLM2s/s3456/Apache%20Superset.jpg" style="margin-left: auto; margin-right: auto;"><img alt="Data Visualization - Apache Superset Guide. Image Source: Unsplash" border="0" data-original-height="2304" data-original-width="3456" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGk5afX58yS8Nf8YceDYmQShsokfr12Q64kjGJDslA4WRUf-1ltlGFx02MXmFvzwpq6A6Yo2xEtedH2zOADDRHtGV9uJqG1w5O1AzOMBYAdb0v8Fxdq-wd0M2MvUFqhDR2UmeRUvWPZ09WcwnTTc9Roh4YsRQ3oY0vXV8hkRcloVMdNn_eSwNd2W8gLM2s/s16000/Apache%20Superset.jpg" title="Data Visualization - Apache Superset Guide. Image Source: Unsplash" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><br /><span style="font-family: Noto Sans; font-size: x-small;">Data Visualization - Apache Superset Guide. Image Source: Unsplash </span></td></tr></tbody></table><br /><p></p><p style="text-align: justify;"><br /></p><p style="text-align: justify;"><span style="font-family: Noto Sans;">Note: This article provides a comprehensive guide on deploying and using Apache Superset on a Linux server. It covers the installation and configuration process, as well as the benefits and features of Superset. While the primary focus is on Superset, we will also explore the broader concepts of business intelligence, data analytics, and visualization.</span></p><p style="text-align: justify;"><span></span></p><a name='more'></a><span style="font-family: Noto Sans;"><br /></span><p></p><h2 style="text-align: justify;"><span style="font-family: Noto Sans;">Introduction to Business Intelligence</span></h2><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span><p style="text-align: justify;"><span style="font-family: Noto Sans;">In today's data-driven world, businesses are constantly seeking ways to gain insights and make informed decisions. This is where Business Intelligence (BI) comes into play. BI refers to the process of collecting, analyzing, and presenting data to help organizations understand their performance, identify trends, and make data-driven decisions. By leveraging BI tools and techniques, businesses can unlock the power of data analytics and visualization to drive growth and gain a competitive edge.</span></p><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span><h2 style="text-align: justify;"><span style="font-family: Noto Sans;">The Role of Data Analytics in Business Intelligence</span></h2><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span><p style="text-align: justify;"><span style="font-family: Noto Sans;">Data analytics is at the core of business intelligence. It involves the exploration, interpretation, and analysis of data to uncover meaningful patterns and insights. With the help of advanced algorithms and statistical models, businesses can extract valuable information from large datasets and gain a deeper understanding of their operations, customers, and market trends.</span></p><p style="text-align: justify;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Noto Sans;">Data analytics can be classified into four main types:</span></p><p style="text-align: justify;"><span style="font-family: Noto Sans;"><br /></span></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: justify;"><span style="font-family: Noto Sans;"><b>Descriptive Analytics:</b> Describes what has happened in the past by summarizing historical data. It provides a snapshot of the current state and helps businesses understand trends and patterns.</span></p><p style="text-align: justify;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Noto Sans;"><b>Diagnostic Analytics: </b>Focuses on identifying the causes of past events and understanding why they occurred. It helps businesses uncover the root causes behind certain outcomes or trends.</span></p><p style="text-align: justify;"><span style="font-family: Noto Sans;"><br /></span></p><div style="text-align: left;"><p style="text-align: justify;"><span style="font-family: Noto Sans;"><b>Predictive Analytics:</b> Uses historical data and statistical models to predict future outcomes and trends. It enables businesses to make proactive decisions and anticipate potential challenges or opportunities.</span></p></div><div style="text-align: left;"><p style="text-align: justify;"><span style="font-family: Noto Sans;"><br /></span></p></div><div><p style="text-align: justify;"><span style="font-family: Noto Sans;"><b>Prescriptive Analytics:</b> Goes beyond predictions and provides recommendations on the best course of action. It uses optimization techniques and simulation models to guide decision-making. </span></p></div></blockquote><p style="text-align: justify;"> </p><p style="text-align: justify;"><span style="font-family: Noto Sans;">By leveraging data analytics, businesses can gain valuable insights into their operations, customer behaviour, market trends, and more. This information can then be used to drive strategic decision-making and optimize business processes.</span></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><p style="text-align: justify;"><br /></p></div></blockquote><div><h4 style="text-align: left;"></h4><h2 style="text-align: justify;"><span style="font-family: Noto Sans;">The Power of Data Visualization in Business Intelligence </span></h2><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span><p style="text-align: justify;"><span style="font-family: Noto Sans; font-weight: normal;">While data analytics provides the foundation for business intelligence, data visualization is the key to unlocking its true potential. Data visualization refers to the graphical representation of data, allowing users to understand complex information at a glance. By presenting data in a visual format, businesses can communicate insights effectively, spot trends, and identify patterns that might have otherwise gone unnoticed. </span></p><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span><p style="text-align: justify;"><span style="font-family: Noto Sans; font-weight: normal;">Data visualization offers several benefits: </span></p><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div style="text-align: justify;"><span style="font-family: Noto Sans;"><b>Improved Data Understanding:</b></span><span style="font-family: Noto Sans;"> Visualizing data makes it easier for users to grasp complex information quickly. By presenting data in a visual format, businesses can enhance data comprehension and promote better decision-making. <br /></span></div></blockquote><div style="text-align: justify;"><span style="font-family: Noto Sans;"><br /></span></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div style="text-align: justify;"><span style="font-family: Noto Sans;"><b>Enhanced Insights: </b></span><span style="font-family: Noto Sans;">Visualization helps users identify trends, patterns, and outliers in data. By visually exploring data, businesses can uncover hidden insights and make data-driven decisions. <br /></span></div></blockquote><div style="text-align: justify;"><span style="font-family: Noto Sans;"><br /></span></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div style="text-align: justify;"><span style="font-family: Noto Sans;"><b>Increased Engagement:</b></span><span style="font-family: Noto Sans;"> Visual representations of data are more engaging and memorable compared to raw numbers or text. By using interactive charts, graphs, and dashboards, businesses can captivate their audience and ensure the message is conveyed effectively. <br /></span></div></blockquote><div style="text-align: justify;"><span style="font-family: Noto Sans;"><br /></span></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div style="text-align: justify;"><span style="font-family: Noto Sans;"><b>Efficient Communication:</b></span><span style="font-family: Noto Sans; font-weight: normal;"> Data visualization simplifies the communication of complex ideas and findings. By presenting data visually, businesses can convey information to stakeholders in a clear and concise manner, fostering collaboration and alignment.<br /></span></div></blockquote><div style="text-align: left;"><div style="text-align: justify;"><span style="font-family: "Noto Sans";"><br /></span></div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div><h2 style="text-align: justify;">Apache Superset: An Introduction</h2><div style="text-align: justify;"><br /></div><p style="text-align: justify;">Apache Superset is a modern data exploration and visualization platform that allows you to create interactive dashboards and charts with a code-free interface. It supports a wide range of data sources, including relational databases, big data platforms, and cloud-based storage solutions.</p><div style="text-align: justify;"><br /></div><h3 style="text-align: justify;">Installing Apache Superset on Linux:</h3><p style="text-align: justify;">Before diving into the installation process, it's essential to ensure that the required dependencies are in place. The installation process can vary depending on the operating system. Let's explore the installation steps for different environments.</p><p style="text-align: justify;"><br /></p><h4 style="text-align: justify;">Debian and Ubuntu</h4> <p style="text-align: justify;">For Debian and Ubuntu Linux distributions, the following command installs the necessary dependencies:</p><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><span style="color: #c0caf5;">sudo</span> <span style="color: #9ece6a;">apt-get</span> <span style="color: #9ece6a;">install</span> <span style="color: #9ece6a;">build-essential</span> <span style="color: #9ece6a;">libssl-dev</span> <span style="color: #9ece6a;">libffi-dev</span> <span style="color: #9ece6a;">python3-dev</span> <span style="color: #9ece6a;">python3-pip</span> <span style="color: #9ece6a;">libsasl2-dev</span> <span style="color: #9ece6a;">libldap2-dev</span> <span style="color: #9ece6a;">default-libmysqlclient-dev</span></div><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><span style="color: #9ece6a;"><br /></span></div></pre></div><div style="text-align: justify;"><br /></div><h4 style="text-align: justify;">Fedora and RHEL-derivative Linux distributions</h4><div style="text-align: justify;"><p>For Fedora and RHEL-derivative Linux distributions, use the yum package manager to install the required packages:</p><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;">
sudo</span> <span style="color: #9ece6a;">yum</span> <span style="color: #9ece6a;">install</span> <span style="color: #9ece6a;">gcc</span> <span style="color: #9ece6a;">gcc-c++</span> <span style="color: #9ece6a;">libffi-devel</span> <span style="color: #9ece6a;">python-devel</span> <span style="color: #9ece6a;">python-pip</span> <span style="color: #9ece6a;">python-wheel</span> <span style="color: #9ece6a;">openssl-devel</span> <span style="color: #9ece6a;">cyrus-sasl-devel</span> <span style="color: #9ece6a;">openldap-devel
</span></div></div></pre></div><div><br /></div></div><p style="text-align: justify;">In more recent versions of CentOS and Fedora, you may need to use <span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;"><span style="color: #0b5394;">dnf</span></span> instead of yum:</p></span><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><span style="color: #c0caf5;">sudo</span> <span style="color: #9ece6a;">dnf</span> <span style="color: #9ece6a;">install</span> <span style="color: #9ece6a;">gcc</span> <span style="color: #9ece6a;">gcc-c++</span> <span style="color: #9ece6a;">libffi-devel</span> <span style="color: #9ece6a;">python3-devel</span> <span style="color: #9ece6a;">python3-pip</span> <span style="color: #9ece6a;">python3-wheel</span> <span style="color: #9ece6a;">openssl-devel</span> <span style="color: #9ece6a;">cyrus-sasl-devel</span> <span style="color: #9ece6a;">openldap-devel</span> </div><span style="color: #9ece6a;">
</span></div></div></pre></div><div><br /></div></div><p style="text-align: justify;">On CentOS, you may also need to upgrade pip:</p><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;">
pip3</span> <span style="color: #9ece6a;">install</span> <span style="color: #e0af68;">--upgrade</span> <span style="color: #9ece6a;">pip
</span></div></div></div></div></pre></div><div><br /></div></div><h4 style="text-align: justify;">Creating a Virtual Environment</h4><div style="text-align: justify;">To ensure a clean and isolated installation, it's highly recommended to install Superset inside a virtual environment. Python provides <span style="background-color: white; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;"><span style="color: #134f5c;">virtualenv</span></span> out of the box, or you can use <span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;"><span style="color: #134f5c;">pyenv</span></span> and<span style="background-color: white; color: #134f5c;"> <span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">pyenv-virtualenv</span></span>. Here's how to create and activate a virtual environment:</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">Using <span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">virtualenv</span>: </div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><div><span style="color: #c0caf5;">pip</span> <span style="color: #9ece6a;">install</span> <span style="color: #9ece6a;">virtualenv</span> </div><div><span style="color: #c0caf5;">virtualenv</span> <span style="color: #9ece6a;">superset</span> </div><div><span style="color: #0db9d7;">source</span> <span style="color: #9ece6a;">superset/bin/activate</span></div></div><span style="color: #9ece6a;">
</span></div></div></div></div></pre></div><div><br /></div></div><p style="text-align: justify;">Once the virtual environment is activated, all Python packages installed or uninstalled will be confined to this environment. You can exit the environment by running deactivate on the command line.</p><div style="text-align: justify;"><br /></div><h4 style="text-align: justify;">Installing Apache Superset</h4><div style="text-align: justify;">To install Apache Superset, use the following command: </div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /></span></div><div><div style="line-height: 20px;"><span style="color: #c0caf5;">pip</span> <span style="color: #9ece6a;">install</span> <span style="color: #9ece6a;">apache-superset</span></div></div><div><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></pre></div><div><br /></div></div><p style="text-align: justify;">After installation, the next step is to initialize the database. However, before running the database upgrade, it is important to configure the user-specified value of <span style="background-color: white;"><span style="color: #134f5c;"><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">SECRET</span><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;"> </span><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">KEY</span></span></span>.</p></span><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div><h4 style="text-align: justify;">Setting up Apache Superset Configuration file</h4><p style="text-align: justify;">Create a directory called superset or any name you prefer:</p><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><span style="color: #c0caf5;">mkdir</span> <span style="color: #9ece6a;">superset</span></div></div></div><div><br /></div></div></div></div></div></div></pre></div></div><div style="text-align: justify;"><br /></div><p style="text-align: justify;">Create a Python file called<span style="color: #134f5c;"> </span><span style="background-color: white; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;"><span style="color: #134f5c;">superset_config.py </span></span>and add the file path to the <span style="background-color: white;"><span style="color: #134f5c;"><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">PYTHONPATH</span> </span></span>environment variable.</p><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><span style="color: #9d7cd8; font-style: italic;">export</span> <span style="color: #c0caf5;">PYTHONPATH</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">"${</span><span style="color: #c0caf5;">PYTHONPATH</span><span style="color: #89ddff;">}</span><span style="color: #9ece6a;">:/visualization/superset</span><span style="color: #89ddff;">"</span></div></div></div></div><div><br /></div></div></div></div></div></div></pre></div><div><br /></div></div></span><span style="font-family: Noto Sans;"><p style="text-align: justify;">To set it permanently append the above line in the shell configuration file <span style="color: #134f5c;"><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">(</span><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">~/.bashrc</span><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">)</span></span> and reload the profile for changes to effect.</p><div style="text-align: justify;"><br /></div><p style="text-align: justify;">After adding this line, you can verify that the <span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">PYTHONPATH</span> environment variable has been set correctly by running the command</p><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #0db9d7;"><br /></span></div><div style="line-height: 20px;"><span style="color: #0db9d7;">echo</span> <span style="color: #c0caf5;">$PYTHONPATH</span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div></div><div style="text-align: justify;"><br /></div><p style="text-align: justify;">Let's add the below file line to configure the <span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">SECRET</span><span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;"> </span><span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">KEY</span> variable.</p><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #0db9d7;"><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><span style="color: #c0caf5;">SECRET_KEY</span> <span style="color: #9ece6a;">=</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">MY_SECRET_KEY</span><span style="color: #89ddff;">'</span></div></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div></div><div style="text-align: justify;"><br /></div><p style="text-align: justify;">After <span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">SECRET</span><span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;"> </span><span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">KEY</span> configuration, the next step is to initialize the database:</p> <div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><span style="color: #c0caf5;">superset</span> <span style="color: #9ece6a;">db</span> <span style="color: #9ece6a;">upgrade</span></div></div></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div></div><p style="text-align: justify;">Once the database is initialized and the configuration is set, you can proceed to create an admin user, load example data, create
default roles and permissions, and build JavaScript assets. These steps ensure a smooth installation and set up the necessary
components for Superset to function properly.</p><div style="text-align: justify;"><br /></div><h4 style="text-align: justify;">Create an Admin User</h4><div style="text-align: justify;"><br /></div><p style="text-align: justify;">To access the Superset web interface, you need to create an admin user account. You can do this by running:</p><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;">superset</span> <span style="color: #9ece6a;">fab</span> <span style="color: #9ece6a;">create-admin</span></div><div style="line-height: 20px;"><span style="color: #9ece6a;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div></div><div style="text-align: justify;"><br /></div><p style="text-align: justify;">This will prompt you to enter your username, email, password, first name, and last name. You can also use any existing user account from your authentication backend by setting the <span style="background-color: white;"><span style="color: #134f5c;"><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">AUTH_TYPE</span> </span></span>environment variable.</p></span><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div><h4 style="text-align: justify;">Load Some Sample Data</h4><div style="text-align: justify;"><br /></div><p style="text-align: justify;">Apache Superset comes with some sample data sets that you can use to explore its features and functionalities. You can load these data sets by running:</p><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;">superset</span> <span style="color: #9ece6a;">load_examples</span></div><div style="line-height: 20px;"><span style="color: #9ece6a;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div></div><div style="text-align: justify;"><br /></div><p style="text-align: justify;">This will load some data sources and dashboards into your Superset instance. You can also add your own data sources by using the SQL Lab or the Data menu in the web interface.</p><div style="text-align: justify;"><br /></div><p style="text-align: justify;">Finally, to start the Superset web server, run the following command:</p><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;">superset</span> <span style="color: #9ece6a;">run</span> <span style="color: #e0af68;">-p</span> <span style="color: #ff9e64;">8088</span> <span style="color: #e0af68;">--with-threads</span> <span style="color: #e0af68;">--reload</span> <span style="color: #e0af68;">--debugger</span></div><div style="line-height: 20px;"><span style="color: #e0af68;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div><div>If everything is set up correctly, navigate to <span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">hostname:port</span> in your browser (e.g., <span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">http://localhost:8080</span>) and log in using the username and password you created.</div></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">Congratulations! You have successfully installed and used Apache Superset on Linux. You can now start creating beautiful visualisations and dashboards with your data.</div><div style="text-align: justify;"><br /></div><h2 style="text-align: justify;">Configuring Apache Superset</h2><div><br /></div></span></div><div style="text-align: left;"><span style="font-family: Noto Sans;"><h3 style="text-align: justify;">Branding Apache Superset</h3><div style="text-align: justify;">To change the Superset application name, logo, favicon, logo target path, logo tooltip and header title set the value for the below variables in the configuration file <span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">superset_config.py</span></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6; line-height: 20px;"><div><div style="line-height: 20px;"><div><span style="color: #7dcfff;">from</span> typing <span style="color: #7dcfff;">import</span> Callable</div><br /><div><span style="color: #444b6a; font-style: italic;"># Uncomment to setup Your App name</span></div><div><span style="color: #e0af68;">APP_NAME</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Superset</span><span style="color: #89ddff;">"</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Specify the App icon</span></div><div><span style="color: #e0af68;">APP_ICON</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">/static/assets/images/superset-logo-horiz.png</span><span style="color: #89ddff;">"</span></div><div><span style="color: #444b6a; font-style: italic;"># replace the image specified in the above path or update your image name</span></div><div><span style="color: #444b6a; font-style: italic;"># file path: superset/lib/python3.11/site-packages/superset/static/assets/images/</span></div><div><span style="color: #444b6a; font-style: italic;"># replace superset with the virtual environment name</span></div><div><span style="color: #444b6a; font-style: italic;"># file path: venv/lib/python3.11/site-packages/superset/static/assets/images/ </span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Specify where clicking the logo would take the user</span></div><div><span style="color: #444b6a; font-style: italic;"># e.g. setting it to '/' would take the user to '/superset/welcome/'</span></div><div><span style="color: #e0af68;">LOGO_TARGET_PATH</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">None</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Specify tooltip that should appear when hovering over the App Icon/Logo</span></div><div><span style="color: #e0af68;">LOGO_TOOLTIP</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">""</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Specify any text that should appear to the right of the logo</span></div><div><span style="color: #e0af68;">LOGO_RIGHT_TEXT</span><span style="color: #89ddff;">:</span> Callable<span style="color: #9abdf5;">[</span><span style="color: #89ddff;">[],</span> <span style="color: #0db9d7;">str</span><span style="color: #9abdf5;">]</span> <span style="color: #bb9af7;">|</span> <span style="color: #0db9d7;">str</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">""</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># Multiple favicons can be specified here. The "href" property</span></div><div><span style="color: #444b6a; font-style: italic;"># is mandatory, but "sizes," "type," and "rel" are optional.</span></div><div><span style="color: #444b6a; font-style: italic;"># For example:</span></div><div><span style="color: #444b6a; font-style: italic;"># {</span></div><div><span style="color: #444b6a; font-style: italic;"># "href":path/to/image.png",</span></div><div><span style="color: #444b6a; font-style: italic;"># "sizes": "16x16",</span></div><div><span style="color: #444b6a; font-style: italic;"># "type": "image/png"</span></div><div><span style="color: #444b6a; font-style: italic;"># "rel": "icon"</span></div><div><span style="color: #444b6a; font-style: italic;"># },</span></div><div><span style="color: #e0af68;">FAVICONS</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">[{</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">href</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">/static/assets/images/favicon.png</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">}]</span> </div><div><span style="color: #444b6a; font-style: italic;"># replace the image specified in the above path or update your image name</span></div><div><span style="color: #444b6a; font-style: italic;"># file path: superset/lib/python3.11/site-packages/superset/static/assets/images/</span></div><div><span style="color: #444b6a; font-style: italic;"># replace superset with the virtual environment name</span></div><div><span style="color: #444b6a; font-style: italic;"># file path: venv/lib/python3.11/site-packages/superset/static/assets/images/ </span></div><div><span style="color: #444b6a; font-style: italic;"><br /></span></div></div></div></div></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div></div><h3 style="text-align: justify;">Setting up MySQL or PostgreSQL as a Production Metastore</h3><div style="text-align: justify;"><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">One of the requirements for installing and running Apache Superset is to have a metadata database that stores information such as user credentials, dashboard configurations, query history, and more. </p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">By default, Superset uses SQLite as the metadata database, which is a simple and lightweight file-based database. However, SQLite has some limitations, such as a lack of concurrency support, scalability issues, and security risks. Therefore, it is recommended to use a more robust and reliable database system for production environments, such as MySQL or PostgreSQL. </p></div><p style="text-align: justify;"><br />To set up the production database:</p><p style="text-align: justify;"></p><div style="text-align: justify;"><br /></div><span style="text-align: start;"><div style="text-align: justify;">Install the database driver for MySQL or PostgreSQL. You can use pip to install the driver package. </div></span><span style="text-align: start;"><div style="text-align: justify;"><br /></div><div style="text-align: justify;">For MySQL, you need to install <span style="background-color: white; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: start; white-space: pre;"><span style="color: #134f5c;">mysqlclient</span></span>:</div><div style="text-align: justify;"><br /></div></span><p></p><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;">pip <span style="color: #9ece6a;">install</span> <span style="color: #9ece6a;">mysqlclient</span></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><span style="color: #9ece6a;"><br /></span></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">For PostgreSQL, you need to install <span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: start; white-space: pre;">psycopg2</span>:</p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6; line-height: 20px;"><span style="color: #c0caf5;">pip</span> <span style="color: #9ece6a;">install</span> <span style="color: #9ece6a;">psycopg2</span></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><span style="color: #9ece6a;"><br /></span></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">Create a database and a user for Superset on your database server. You can use any tool or command line interface to do this. </p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">For example, for MySQL, you can use the following commands:</p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6; line-height: 20px;"><div><span style="color: #c0caf5;">mysql</span> <span style="color: #e0af68;">-u</span> <span style="color: #9ece6a;">root</span> <span style="color: #e0af68;">-p</span></div><br /><div><span style="color: #c0caf5;">CREATE</span> <span style="color: #9ece6a;">DATABASE</span> <span style="color: #9ece6a;">superset</span><span style="color: #89ddff;">;</span></div><br /><div><span style="color: #c0caf5;">CREATE</span> <span style="color: #9ece6a;">USER</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">superset</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">@</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">localhost</span><span style="color: #89ddff;">'</span> <span style="color: #9ece6a;">IDENTIFIED</span> <span style="color: #9ece6a;">BY</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">superset</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">;</span></div><br /><div><span style="color: #c0caf5;">GRANT</span> <span style="color: #9ece6a;">ALL</span> <span style="color: #9ece6a;">PRIVILEGES</span> <span style="color: #9ece6a;">ON</span> <span style="color: #9ece6a;">superset.</span><span style="color: #f7768e;">*</span> <span style="color: #9ece6a;">TO</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">superset</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">@</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">localhost</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">;</span></div></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div><div><br /></div><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">For PostgreSQL, you can use the following commands:</p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div><span style="color: #c0caf5;">psql</span> <span style="color: #e0af68;">-U</span> <span style="color: #9ece6a;">postgres</span></div><br /><div><span style="color: #c0caf5;">CREATE</span> <span style="color: #9ece6a;">DATABASE</span> <span style="color: #9ece6a;">superset</span><span style="color: #89ddff;">;</span></div><br /><div><span style="color: #c0caf5;">CREATE</span> <span style="color: #9ece6a;">USER</span> <span style="color: #9ece6a;">superset</span> <span style="color: #9ece6a;">WITH</span> <span style="color: #9ece6a;">PASSWORD</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">superset</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">;</span></div><br /><div><span style="color: #c0caf5;">GRANT</span> <span style="color: #9ece6a;">ALL</span> <span style="color: #9ece6a;">PRIVILEGES</span> <span style="color: #9ece6a;">ON</span> <span style="color: #9ece6a;">DATABASE</span> <span style="color: #9ece6a;">superset</span> <span style="color: #9ece6a;">TO</span> <span style="color: #9ece6a;">superset</span><span style="color: #89ddff;">;</span></div></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">Edit the Superset configuration file <span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: left; white-space: pre;">superset_config.py </span>to specify the database URI for the metadata database.</p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">In the configuration file, look for the line that starts with <span style="background-color: white;"><span style="color: #134f5c;"><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">SQLALCHEMY_DATABASE_URI</span><span style="font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;"> </span></span></span> and change it to point to your MySQL or PostgreSQL database. </p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">The format of the URI is:</p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6; line-height: 20px;">dialect<span style="color: #89ddff;">+</span>driver<span style="color: #89ddff;">://</span>username<span style="color: #89ddff;">:</span>password<span style="color: #89ddff;">@</span>host<span style="color: #89ddff;">:</span>port<span style="color: #89ddff;">/</span>database</div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><span style="color: #9ece6a;"><br /></span></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">For example, for MySQL, you can use:</p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><br class="Apple-interchange-newline" /><div style="line-height: 20px;"><span style="color: #e0af68;">SQLALCHEMY_DATABASE_URI</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">mysql+mysqldb://superset:superset@localhost:3306/superset</span><span style="color: #89ddff;">'</span></div></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><span style="color: #9ece6a;"><br /></span></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;">For PostgreSQL, you can use:</p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #e0af68;">
SQLALCHEMY_DATABASE_URI</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">postgresql+psycopg2://superset:superset@localhost:5432/superset</span><span style="color: #89ddff;">'</span></div><div style="line-height: 20px;"><span style="color: #89ddff;"><br /></span></div></div></div></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><br /></div><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">Initialize the metadata database by running the following command:</p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;">superset</span> <span style="color: #9ece6a;">db</span> <span style="color: #9ece6a;">upgrade</span></div><div style="line-height: 20px;"><span style="color: #9ece6a;"><br /></span></div></div></div></div></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">This will create the necessary tables and indexes for Superset in your database.</p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: justify;">Restart Superset by running the following command</p></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><span style="color: #c0caf5;">superset</span><span style="color: #a9b1d6;"> </span><span style="color: #9ece6a;">run</span><span style="color: #a9b1d6;"> </span><span style="color: #e0af68;">-p</span><span style="color: #a9b1d6;"> </span><span style="color: #ff9e64;">8088</span><span style="color: #a9b1d6;"> </span><span style="color: #e0af68;">--with-threads</span><span style="color: #a9b1d6;"> </span><span style="color: #e0af68;">--reload</span><span style="color: #a9b1d6;"> </span><span style="color: #e0af68;">--debugger</span></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><h3 style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px;">Email Integration</h3><div><br /></div><div><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: start;">Superset can be configured to send email alerts when a SQL condition is reached and schedule reports to send screenshots of dashboards and charts.</p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: start;"><br /></p><p class="p1" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; text-align: start;">To enable the alerts & reporting feature, update the <span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">superset_config.py</span> file as follows:</p></div></div><div style="text-align: justify;"><br /></div></span><div><span style="font-family: Noto Sans;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: start; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6; line-height: 20px;"><div><span style="color: #e0af68;">FEATURE_FLAGS</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">ALERT_REPORTS</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">ALERTS_ATTACH_REPORTS</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span></div><div><span style="color: #89ddff;">}</span></div></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px;">Superset uses Celery Beat as a scheduler and Celery worker for sending alerts and reports.</p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p></span></div></div><span style="font-family: "Noto Sans";"></span><blockquote><span style="font-family: "Noto Sans";">Celery</span><br /><span style="font-family: Noto Sans;"><br /></span><span style="font-family: "Noto Sans";">Celery is an open-source distributed task queue framework. It allows you to run tasks asynchronously (for example sending an email) and distribute them across multiple workers. Ideal for background processing and task scheduling, etc.</span><br /><span style="font-family: Noto Sans;"><br /></span><span style="font-family: "Noto Sans";">Celery Beat:</span><br /><span style="font-family: Noto Sans;"><br /></span><span style="font-family: "Noto Sans";">Celery beat is the scheduling component of Celery, responsible for managing periodic or scheduled tasks. The schedule information can be stored in a different backend such as a database or an in-memory store.</span><br /><span style="font-family: Noto Sans;"><br /></span><span style="font-family: "Noto Sans";">Celery Worker:</span><br /><span style="font-family: Noto Sans;"><br /></span><div style="text-align: left;"><div><span style="font-family: Noto Sans;"><div style="text-align: left;">The celery worker is responsible for executing tasks that are enqueued by the Celery application. When you define and enqueue tasks in the application. It is added to a message queue (such as RabbitMQ, Redis or others) and Celery workers pull tasks from the queue and execute them. Workers are usually distributed across machines or processes, enabling you to parallelise the execution of tasks and achieve better performance and scalability.</div></span></div></div></blockquote><div style="text-align: left;"><div><span style="font-family: Noto Sans;"><div style="text-align: left;"></div></span></div></div><div style="text-align: left;"><div><span style="font-family: Noto Sans;"><div><br /></div><p style="text-align: left;">Celery Configuration:</p><div><br /></div><div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #7dcfff;">from</span> celery<span style="color: #89ddff;">.</span>schedules <span style="color: #7dcfff;">import</span> crontab</div><br /><div><span style="color: #444b6a; font-style: italic;"># Celery configuration</span></div><div><span style="color: #e0af68;">REDIS_HOST</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">localhost</span><span style="color: #89ddff;">"</span></div><div><span style="color: #e0af68;">REDIS_PORT</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">6379</span><span style="color: #89ddff;">"</span></div><br /><div><span style="color: #bb9af7;">class</span> <span style="color: #c0caf5;">CeleryConfig</span><span style="color: #9abdf5;">:</span></div><div> broker_url <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">redis://</span><span style="color: #bb9af7;">%s</span><span style="color: #9ece6a;">:</span><span style="color: #bb9af7;">%s</span><span style="color: #9ece6a;">/0</span><span style="color: #89ddff;">'</span> <span style="color: #89ddff;">%</span> <span style="color: #89ddff;">(</span><span style="color: #e0af68;">REDIS_HOST</span><span style="color: #89ddff;">,</span> <span style="color: #e0af68;">REDIS_PORT</span><span style="color: #89ddff;">)</span></div><div> imports <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">(</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">superset.sql_lab</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">superset.tasks</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">superset.tasks.thumbnails</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span> <span style="color: #89ddff;">)</span></div><div> result_backend <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">redis://</span><span style="color: #bb9af7;">%s</span><span style="color: #9ece6a;">:</span><span style="color: #bb9af7;">%s</span><span style="color: #9ece6a;">/0</span><span style="color: #89ddff;">'</span> <span style="color: #89ddff;">%</span> <span style="color: #89ddff;">(</span><span style="color: #e0af68;">REDIS_HOST</span><span style="color: #89ddff;">,</span> <span style="color: #e0af68;">REDIS_PORT</span><span style="color: #89ddff;">)</span></div><div> worker_prefetch_multiplier <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">10</span></div><div> task_acks_late <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">True</span></div><div> task_annotations <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">sql_lab.get_sql_results</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">rate_limit</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">100/s</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">},</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">email_reports.send</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">rate_limit</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">1/s</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">time_limit</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">600</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">soft_time_limit</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">600</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">ignore_result</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">},</span></div><div> <span style="color: #89ddff;">}</span></div><div> beat_schedule <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">reports.scheduler</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">task</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">reports.scheduler</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">schedule</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #7aa2f7;">crontab</span><span style="color: #9abdf5;">(</span><span style="color: #e0af68;">minute</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">*</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">hour</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">*</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">},</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">reports.prune_log</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">task</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">reports.prune_log</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">schedule</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #7aa2f7;">crontab</span><span style="color: #9abdf5;">(</span><span style="color: #e0af68;">minute</span><span style="color: #89ddff;">=</span><span style="color: #ff9e64;">0</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">hour</span><span style="color: #89ddff;">=</span><span style="color: #ff9e64;">0</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">},</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">email_reports.schedule_hourly</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">task</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">email_reports.schedule_hourly</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">schedule</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #7aa2f7;">crontab</span><span style="color: #9abdf5;">(</span><span style="color: #e0af68;">minute</span><span style="color: #89ddff;">=</span><span style="color: #ff9e64;">1</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">hour</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">*</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">},</span></div><div> <span style="color: #89ddff;">}</span></div><br /><div><span style="color: #e0af68;">CELERY_CONFIG</span> <span style="color: #89ddff;">=</span> CeleryConfig</div></div></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: left;">SMTP Configuration:</p></div><div><br /></div><div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><span style="color: #e0af68;"><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><div><span style="color: #e0af68;">SMTP_HOST</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">smtp.mydomain.com</span><span style="color: #89ddff;">"</span> <span style="color: #444b6a; font-style: italic;"># change to your host</span></div><div><span style="color: #e0af68;">SMTP_PORT</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">25</span> <span style="color: #444b6a; font-style: italic;"># your port, e.g. 587</span></div><div><span style="color: #e0af68;">SMTP_STARTTLS</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">True</span></div><div><span style="color: #e0af68;">SMTP_SSL_SERVER_AUTH</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">False</span> <span style="color: #444b6a; font-style: italic;"># If your using an SMTP server with a valid certificate</span></div><div><span style="color: #e0af68;">SMTP_SSL</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">False</span></div><div><span style="color: #e0af68;">SMTP_USER</span> <span style="color: #89ddff;">=</span> <span style="color: #e0af68;">SMTP_USER</span> <span style="color: #444b6a; font-style: italic;"># use the empty string "" if using an unauthenticated SMTP server</span></div><div><span style="color: #e0af68;">SMTP_PASSWORD</span> <span style="color: #89ddff;">=</span> <span style="color: #e0af68;">SMTP_PASSWORD</span> <span style="color: #444b6a; font-style: italic;"># use the empty string "" if using an unauthenticated SMTP server</span></div><div><span style="color: #e0af68;">SMTP_MAIL_FROM</span> <span style="color: #89ddff;">=</span> <span style="color: #e0af68;">SMTP_USER</span> </div><div><span style="color: #e0af68;"><br /></span></div><div><span style="color: #e0af68;">EMAIL_REPORTS_SUBJECT_PREFIX</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Insights - </span><span style="color: #89ddff;">"</span> <span style="color: #444b6a; font-style: italic;"># optional - overwrites default value in config.py of "[Report] "</span></div></div></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div><span style="color: #444b6a; font-style: italic;"># The text for call-to-action link in Alerts & Reports emails</span></div><div><span style="color: #e0af68;">EMAIL_REPORTS_CTA</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Explore in BI Portal</span><span style="color: #89ddff;">"</span></div><div><span style="color: #e0af68;">ALERT_REPORTS_NOTIFICATION_DRY_RUN</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">False</span></div></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p></div><p style="text-align: left;">Screenshot Configuration:</p><div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><span style="color: #e0af68;"><br class="Apple-interchange-newline" /></span><div style="line-height: 20px;"><div><span style="color: #444b6a; font-style: italic;"># WebDriver configuration</span></div><div><span style="color: #e0af68;">WEBDRIVER_TYPE</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">chrome</span><span style="color: #89ddff;">"</span></div><br /><div><span style="color: #e0af68;">WEBDRIVER_OPTION_ARGS</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">[</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">--force-device-scale-factor=2.0</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">--high-dpi-support=2.0</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">--headless</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">--disable-gpu</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">--disable-dev-shm-usage</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">--no-sandbox</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">--disable-setuid-sandbox</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">--disable-extensions</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div><span style="color: #89ddff;">]</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># This is for internal use, you can keep http</span></div><div><span style="color: #e0af68;">WEBDRIVER_BASEURL</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">http://127.0.0.1:8000</span><span style="color: #89ddff;">"</span></div><br /><div><span style="color: #444b6a; font-style: italic;"># This is the link sent to the recipient. Change to your domain, e.g. https://superset.mydomain.com</span></div><div><span style="color: #e0af68;">WEBDRIVER_BASEURL_USER_FRIENDLY</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">https://superset.mydomain.com</span><span style="color: #89ddff;">"
</span><div style="line-height: 20px;"><div><span style="color: #e0af68;">SCREENSHOT_LOCATE_WAIT</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">100</span></div><div><span style="color: #e0af68;">SCREENSHOT_LOAD_WAIT</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">600</span></div></div><span style="color: #89ddff;">
</span></div></div></div></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p><p style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: left;">Generic configuration:</p></div><div><br /></div><div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><div><div style="line-height: 20px;"><div><span style="color: #444b6a; font-style: italic;">
# Execute Alerts & Reports as admin User</span></div><div><span style="color: #e0af68;">THUMBNAIL_SELENIUM_USER</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">admin</span><span style="color: #89ddff;">'</span></div><div><span style="color: #e0af68;">ALERT_REPORTS_EXECUTE_AS</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">[</span>ExecutorType<span style="color: #89ddff;">.</span><span style="color: #e0af68;">SELENIUM</span><span style="color: #89ddff;">]
</span></div></div></div></div></div></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px;"><br /></p></div><div><h3 style="text-align: justify;">Embedding Apache Superset Dashboards</h3></div><div>One of its powerful features is the ability to embed dashboards within external applications, providing seamless integration and enhanced data analysis capabilities.<br /><br />To begin embedding Superset dashboards, follow these steps:<br /><br /><br /><b>Enable Embedded Superset Feature:</b></div><div><ul><li><span style="font-family: Noto Sans;">Edit the Feature Flag configuration by adding the Embedded Superset flag:</span></li></ul></div><div><br /></div><div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #444b6a; font-style: italic;"><div style="color: #a9b1d6; font-style: normal;"><span style="color: #ff9e64;">
FEATURE_FLAGS</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span></div><div style="color: #a9b1d6; font-style: normal;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">EMBEDDED_SUPERSET</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6; font-style: normal;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">DASHBOARD_RBAC</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span></div><span style="color: black; font-family: Noto Sans; font-size: medium; font-style: normal; white-space: normal;"><span style="color: #89ddff; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">}</span></span></span></div><div><br /></div></div></div></div></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div></div><div><br /><ul><li>Restart the Superset instance to activate the Embedded Superset feature.</li></ul><div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div><span style="color: #444b6a; font-style: italic;"><div style="color: #a9b1d6; font-style: normal;"><span style="color: #ff9e64;"><span style="color: #c0caf5;">
superset</span><span style="color: #a9b1d6;"> </span><span style="color: #9ece6a;">run</span><span style="color: #a9b1d6;"> </span><span style="color: #e0af68;">-p</span><span style="color: #a9b1d6;"> </span><span style="color: #ff9e64;">8088</span><span style="color: #a9b1d6;"> </span><span style="color: #e0af68;">--with-threads</span><span style="color: #a9b1d6;"> </span><span style="color: #e0af68;">--reload</span><span style="color: #a9b1d6;"> </span><span style="color: #e0af68;">--debugger
</span></span></div></span></div></div></div></div></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><div><ul></ul></div></div></div><div><br /></div><div><b>Dashboard Configuration</b></div><div><ul><li><span style="font-family: Noto Sans;">Access the dashboard sub-menu and click on "Embed Dashboard."</span></li><li><span style="font-family: Noto Sans;">Enable Embedding and make a note of the generated embedding ID.</span></li></ul><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfFMm954xTzeucNrj1vrOaPQhbldpundrTpwuUAcI0esLPwatkabJaDANPgMu2uqhT4ealAVO6ZFHDlYBK1P8sq4hgxfo9ZazMtzg9BpNeN0k8d5X_LSsaEinIibsvAzYMpapf_f_TBFJVCzHCjhi-_vF_-uVaHu_0QZIRnS14ZHy11-qb01xlj9HD8vN3/s2226/image.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="Apache Superset Embedded Settings" border="0" data-original-height="1186" data-original-width="2226" height="341" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfFMm954xTzeucNrj1vrOaPQhbldpundrTpwuUAcI0esLPwatkabJaDANPgMu2uqhT4ealAVO6ZFHDlYBK1P8sq4hgxfo9ZazMtzg9BpNeN0k8d5X_LSsaEinIibsvAzYMpapf_f_TBFJVCzHCjhi-_vF_-uVaHu_0QZIRnS14ZHy11-qb01xlj9HD8vN3/w640-h341/image.png" title="Apache Superset Embedded Settings" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Apache Superset Embedded Settings</td></tr></tbody></table><br /><div><b>Creating a dedicated user for Embedded Dashboard access</b><br /><ul><li><span style="font-family: Noto Sans;">Go to Settings and select "List User."</span></li><li><span style="font-family: Noto Sans;">Add a new user record dedicated to embedded dashboard access.</span></li><li><span style="font-family: Noto Sans;">Enable restricted access for the user.</span></li></ul><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjt9Eo2Qwc1giLIdmBQC8Vo5uBxZDGSPUkQevNR8Ai08vGaMCVdhZCe56dk8fRw57NmLrFFeupA37iG3zCyTHXItQImbbDik5VBS_NttEoMZytZ3ZPb-baDOXsDAGP-QyIes9JIiJ7Z3fai5ZtzuI5xxoPhQtR1TIfi2UjVHJ58ifvZ4Ycfu-y2T3CrNKIX/s2294/adding%20user.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="Apache Superset adding an user" border="0" data-original-height="1120" data-original-width="2294" height="312" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjt9Eo2Qwc1giLIdmBQC8Vo5uBxZDGSPUkQevNR8Ai08vGaMCVdhZCe56dk8fRw57NmLrFFeupA37iG3zCyTHXItQImbbDik5VBS_NttEoMZytZ3ZPb-baDOXsDAGP-QyIes9JIiJ7Z3fai5ZtzuI5xxoPhQtR1TIfi2UjVHJ58ifvZ4Ycfu-y2T3CrNKIX/w640-h312/adding%20user.png" title="Apache Superset adding an user" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Apache Superset adding an user</td></tr></tbody></table><br /><div><br /></div></div></div><div><b>Generating Guest Token</b></div><div><br /></div><div>Apache Superset uses a access token called guest token to enable access to your user without requiring login authentication from the hosted application. You can generate guest token using Superset API guest token endpoint. </div><div><br /></div><div>First generate login access token using /login endpoint by passing the above created user credentials: </div><div><br /></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYZG6g4V25VPvJBJW5yS2YpMM3-MYElsezfC4f6qczlIhEYLW22yNn5_93B7HdGywzaA7j5vbGW00ZOjlZSt0BiIF7r-XlaopPfqQIWBtXuqfW8LIgdTCbVyJCv3vSZxNEX8VkGI-yHVHwpSGfUNiZUz37w-b7XAweIc4kntV9THCjCCTAUPanwFmavGlF/s2298/login%20access%20token.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="Apache Superset API - Generating login token" border="0" data-original-height="1048" data-original-width="2298" height="292" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYZG6g4V25VPvJBJW5yS2YpMM3-MYElsezfC4f6qczlIhEYLW22yNn5_93B7HdGywzaA7j5vbGW00ZOjlZSt0BiIF7r-XlaopPfqQIWBtXuqfW8LIgdTCbVyJCv3vSZxNEX8VkGI-yHVHwpSGfUNiZUz37w-b7XAweIc4kntV9THCjCCTAUPanwFmavGlF/w640-h292/login%20access%20token.png" title="Apache Superset API - Generating login token" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Apache Superset API - Generating login token</td></tr></tbody></table><div><br /></div><div>Now pass the login access token along with guest token parameters to /guest_token endpoint to generate guest token.</div><div><br /></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhB7wTkZT9hbdpB1_fHWe5CbB89CJ3PuyAEOyaWZAuj6EJGxM7uGMtP1zj4bFoRIT7lMedRBoVEY-zD8K1XOvg3XPb0eoQ5EPdCq9kJ7cPnpFcP8AIWO4tc1imdtaeA3HC276n_Ze4NE7Xav6sz_tClNmmgfFY4xVWPsZdRacg1aMv0KPsDC39nni4vjptk/s2284/image.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="Apache Superset API - Generating guest token" border="0" data-original-height="1486" data-original-width="2284" height="416" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhB7wTkZT9hbdpB1_fHWe5CbB89CJ3PuyAEOyaWZAuj6EJGxM7uGMtP1zj4bFoRIT7lMedRBoVEY-zD8K1XOvg3XPb0eoQ5EPdCq9kJ7cPnpFcP8AIWO4tc1imdtaeA3HC276n_Ze4NE7Xav6sz_tClNmmgfFY4xVWPsZdRacg1aMv0KPsDC39nni4vjptk/w640-h416/image.png" title="Apache Superset API - Generating guest token" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Apache Superset API - Generating guest token</td></tr></tbody></table><br /><div>You can test this process by using the swagger interface provided by the apache superset. Swagger can be accessed by visiting the <span style="background-color: white; color: #134f5c; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; text-align: justify; white-space: pre;">https://superset.mydomain.com/swagger/v1 </span>path.</div><div><br /></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNgakd0RZ9ynMPejvCwHf8GGy_yigMoNf058fY8mjYqCZof1fRkBz_5Az6DUyUNEEohpKYC5IbwQXxpwoWpqcbmXOVxOjq3pf3wwig-GRP-57nkgza6Qfa9699fCPZ9cHk_9aKGHYvxe29NaafQSdqk8lY2IxDCLWvazM3PGRYAFkkg-QbfGkTkC19nxzf/s2316/Swagger.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="594" data-original-width="2316" height="164" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNgakd0RZ9ynMPejvCwHf8GGy_yigMoNf058fY8mjYqCZof1fRkBz_5Az6DUyUNEEohpKYC5IbwQXxpwoWpqcbmXOVxOjq3pf3wwig-GRP-57nkgza6Qfa9699fCPZ9cHk_9aKGHYvxe29NaafQSdqk8lY2IxDCLWvazM3PGRYAFkkg-QbfGkTkC19nxzf/w640-h164/Swagger.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Apache Superset API - Swagger</td></tr></tbody></table><br /><div><br /></div><div><b>Python code example:</b></div><div><b><br /></b></div><div>Generating login token</div><div><br /></div><div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6;"><span style="color: #bb9af7;">def</span> <span style="color: #7aa2f7;">get_login_token</span><span style="color: #9abdf5;">():</span></div><div style="color: #a9b1d6;"> url <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">https://superset.mydomain.com/api/v1/security/login</span><span style="color: #89ddff;">'</span></div><div style="color: #a9b1d6;"> headers <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span></div><div style="color: #a9b1d6;"><span style="color: #89ddff;"><span> </span><span> </span>'</span><span style="color: #9ece6a;">accept</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">application/json</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span> </div><div style="color: #a9b1d6;"><span style="color: #89ddff;"><span> </span><span> </span>'</span><span style="color: #9ece6a;">Content-Type</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">application/json</span><span style="color: #89ddff;">'</span></div><div style="color: #a9b1d6;"><span style="color: #89ddff;"> }</span></div><div style="color: #a9b1d6;"> data <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">password</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">guestpwd</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">provider</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">db</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">refresh</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">true</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">username</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">guest</span><span style="color: #89ddff;">"</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">}</span></div><div style="color: #a9b1d6;"> session <span style="color: #89ddff;">=</span> requests<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">Session</span><span style="color: #9abdf5;">()</span></div><div style="color: #a9b1d6;"> response <span style="color: #89ddff;">=</span> session<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">post</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">url</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #e0af68;">headers</span><span style="color: #89ddff;">=</span><span style="color: #c0caf5;">headers</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #e0af68;">data</span><span style="color: #89ddff;">=</span><span style="color: #c0caf5;">json</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">dumps</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">data</span><span style="color: #9abdf5;">))</span></div><div style="color: #a9b1d6;"> <span style="color: #bb9af7;">return</span> response<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">json</span><span style="color: #9abdf5;">()</span></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p></div><div>Generating guest token</div><div><br /></div><div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6;"><span style="color: #bb9af7;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6;"><span style="color: #bb9af7;">def</span> <span style="color: #7aa2f7;">get_guest_token</span><span style="color: #9abdf5;">(</span><span style="color: #e0af68;">access_token</span><span style="color: #9abdf5;">):</span></div><div style="color: #a9b1d6;"> url <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">https://superset.mydomain.com/api/v1/security/login</span><span style="color: #89ddff;">'</span></div><div style="color: #a9b1d6;"> headers <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">accept</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">application/json</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">Authorization</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #bb9af7;">f</span><span style="color: #9ece6a;">'Bearer </span><span style="color: #bb9af7;">{</span><span style="color: #c0caf5;">access_token</span><span style="color: #bb9af7;">}</span><span style="color: #9ece6a;">'</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">Content-Type</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">application/json</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">}</span></div><div style="color: #a9b1d6;"> data <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">resources</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">[{</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">id</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">11</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">type</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">dashboard</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">}],</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">rls</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">[],</span></div><div style="color: black;"><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">user</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">{</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">first_name</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">guest</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">last_name</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">user</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">username</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">guest</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">}</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">}</span></div><div style="color: #a9b1d6;"> session <span style="color: #89ddff;">=</span> requests<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">Session</span><span style="color: #9abdf5;">()</span></div><div style="color: #a9b1d6;"> response <span style="color: #89ddff;">=</span> session<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">post</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">url</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #e0af68;">headers</span><span style="color: #89ddff;">=</span><span style="color: #c0caf5;">headers</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #e0af68;">data</span><span style="color: #89ddff;">=</span><span style="color: #c0caf5;">json</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">dumps</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">data</span><span style="color: #9abdf5;">))</span></div><div style="color: #a9b1d6;"> <span style="color: #bb9af7;">return</span> response<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">json</span><span style="color: #9abdf5;">()</span></div></span></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p></div><div>Note: Ensure to handle token refresh within your hosting application / application backend, as the guest token is valid for 5 minutes.</div><div><br /></div><div><b>Embedding Dashboard using Superset Embedded SDK</b></div><div><b><br /></b></div><div><ul style="text-align: left;"><li>Include the Superset Embedded SDK script using a CDN in the <head> section of your HTML page.</li><li>Use the Superset Embedded SDK to embed the dashboard within an <iframe> element.</li></ul><div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6;"><span style="color: #bb9af7;"><div style="color: #a9b1d6;"><span style="color: #bb9af7;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6;"><span style="color: #89ddff;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">script</span><span style="color: #ba3c97;">></span></div><span style="color: #a9b1d6;"><br /></span><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> </span><span style="color: #9d7cd8; font-style: italic;">async</span><span style="color: #c0caf5;"> </span><span style="color: #bb9af7;">function</span><span style="color: #c0caf5;"> </span><span style="color: #7aa2f7;">fetchGuestTokenFromBackend</span><span style="color: #9abdf5;">()</span><span style="color: #c0caf5;"> </span><span style="color: #9abdf5;">{</span></div><div style="color: #a9b1d6;"><span style="color: #89ddff;"> </span><span style="color: #444b6a; font-style: italic;">// let response = await fetch('https://mybackend.com/fetchGuestToken', { method: 'POST'});</span></div><div style="color: black;"><span style="color: #9abdf5;"> </span><span style="color: #9d7cd8; font-style: italic;">let</span><span style="color: #9abdf5;"> </span><span style="color: #bb9af7;">data</span><span style="color: #9abdf5;"> </span><span style="color: #89ddff;">=</span><span style="color: #9abdf5;"> </span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">guest_token_value_generated_using_above_python_code</span><span style="color: #89ddff;">'</span></div><div style="color: #a9b1d6;"><span style="color: #9abdf5;"> </span><span style="color: #bb9af7; font-style: italic;">return</span><span style="color: #9abdf5;"> </span><span style="color: #c0caf5;">data</span><span style="color: #9abdf5;"> </span></div><div style="color: #a9b1d6;"><span style="color: #9abdf5;"> }</span></div><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> supersetEmbeddedSdk</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">embedDashboard</span><span style="color: #9abdf5;">({</span></div><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> </span><span style="color: #73daca;">id</span><span style="color: #89ddff;">:</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">164104a5-69a8-484b-bb51-2a5cf7cb4a29</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #444b6a; font-style: italic;">// given by the Superset embedding UI</span></div><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> </span><span style="color: #73daca;">supersetDomain</span><span style="color: #89ddff;">:</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">https://superset.mydomain.com</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> </span><span style="color: #73daca;">mountPoint</span><span style="color: #89ddff;">:</span><span style="color: #c0caf5;"> document</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">getElementById</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">dashboard-container</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #444b6a; font-style: italic;">// any html element that can contain an iframe</span></div><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> </span><span style="color: #7aa2f7;">fetchGuestToken</span><span style="color: #89ddff;">:</span><span style="color: #c0caf5;"> </span><span style="color: #9abdf5;">()</span><span style="color: #c0caf5;"> </span><span style="color: #bb9af7;">=></span><span style="color: #c0caf5;"> </span><span style="color: #7aa2f7;">fetchGuestTokenFromBackend</span><span style="color: #9abdf5;">()</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> </span><span style="color: #73daca;">dashboardUiConfig</span><span style="color: #89ddff;">:</span><span style="color: #c0caf5;"> </span><span style="color: #9abdf5;">{</span></div><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> </span><span style="color: #73daca;">hideTitle</span><span style="color: #89ddff;">:</span><span style="color: #ff9e64;">true</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> </span><span style="color: #73daca;">hideTab</span><span style="color: #89ddff;">:</span><span style="color: #ff9e64;">true</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> </span><span style="color: #73daca;">hideChartControl</span><span style="color: #89ddff;">:</span><span style="color: #ff9e64;">true</span></div><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> </span><span style="color: #9abdf5;">}</span><span style="color: #c0caf5;"> </span><span style="color: #444b6a; font-style: italic;">// dashboard UI config: hideTitle, hideTab, hideChartControls (optional)</span></div><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> </span><span style="color: #9abdf5;">})</span></div><div style="color: #a9b1d6;"><span style="color: #9abdf5;"><br /></span></div><div style="color: #a9b1d6;"><span style="color: #c0caf5;"> </span><span style="color: #ba3c97;"></</span><span style="color: #f7768e;">script</span><span style="color: #ba3c97;">></span></div></span></div></span></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="text-align: left;">Refer to the provided HTML and JavaScript code examples for the complete embedding process.</span></p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="text-align: left;"><br /></span></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6;"><span style="color: #bb9af7;"><div style="color: #a9b1d6;"><span style="color: #bb9af7;"><div style="color: #a9b1d6;"><span style="color: #89ddff;"><div style="color: #a9b1d6; line-height: 20px;"><div><span style="color: #ba3c97;">
<</span><span style="color: #f7768e;">html</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">head</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">meta</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">http-equiv</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;">Content-Security-Policy</span><span style="color: #89ddff;">"</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">content</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;">upgrade-insecure-requests</span><span style="color: #89ddff;">"</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">title</span><span style="color: #ba3c97;">></span><span style="color: #9aa5ce;">Superset Embedded Example</span><span style="color: #ba3c97;"></</span><span style="color: #f7768e;">title</span><span style="color: #ba3c97;">></span></div><div><span style="color: #89ddff;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">script</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">src</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;">https://unpkg.com/@superset-ui/embedded-sdk</span><span style="color: #89ddff;">"</span><span style="color: #ba3c97;">></</span><span style="color: #f7768e;">script</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">link</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">rel</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;">preconnect</span><span style="color: #89ddff;">"</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">href</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;">https://fonts.googleapis.com</span><span style="color: #89ddff;">"</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">link</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">rel</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;">preconnect</span><span style="color: #89ddff;">"</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">href</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;">https://fonts.gstatic.com</span><span style="color: #89ddff;">"</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">crossorigin</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">link</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">href</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;">https://fonts.googleapis.com/css2?family=Noto+Sans:wght@400;700&display=swap</span><span style="color: #89ddff;">"</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">rel</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;">stylesheet</span><span style="color: #89ddff;">"</span><span style="color: #ba3c97;">></span></div><div><span style="color: #89ddff;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">style</span><span style="color: #ba3c97;">></span></div><div><span style="color: #c0caf5;"> </span><span style="color: #0db9d7;">iframe</span><span style="color: #c0caf5;"> </span><span style="color: #9abdf5;">{</span></div><div><span style="color: #9abdf5;"> </span><span style="color: #7aa2f7;">width</span><span style="color: #89ddff;">:</span><span style="color: #9abdf5;"> </span><span style="color: #ff9e64;">100</span><span style="color: #f7768e;">%</span><span style="color: #89ddff;">;</span></div><div><span style="color: #9abdf5;"> </span><span style="color: #7aa2f7;">height</span><span style="color: #89ddff;">:</span><span style="color: #9abdf5;"> </span><span style="color: #ff9e64;">100</span><span style="color: #f7768e;">%</span><span style="color: #89ddff;">;</span></div><div><span style="color: #9abdf5;"> </span><span style="color: #7aa2f7;">border</span><span style="color: #89ddff;">:</span><span style="color: #9abdf5;"> </span><span style="color: #ff9e64;">none</span><span style="color: #89ddff;">;</span></div><div><span style="color: #9abdf5;"> </span><span style="color: #7aa2f7;">margin-top</span><span style="color: #89ddff;">:</span><span style="color: #9abdf5;"> </span><span style="color: #ff9e64;">3</span><span style="color: #f7768e;">%</span><span style="color: #89ddff;">;</span></div><div><span style="color: #9abdf5;"> }</span></div><br /><div><span style="color: #c0caf5;"> pretext </span><span style="color: #9abdf5;">{</span></div><div><span style="color: #9abdf5;"> </span><span style="color: #7aa2f7;">margin-right</span><span style="color: #89ddff;">:</span><span style="color: #9abdf5;"> </span><span style="color: #ff9e64;">10</span><span style="color: #f7768e;">%</span><span style="color: #89ddff;">;</span></div><div><span style="color: #9abdf5;"> </span><span style="color: #7aa2f7;">margin-left</span><span style="color: #89ddff;">:</span><span style="color: #9abdf5;"> </span><span style="color: #ff9e64;">10</span><span style="color: #f7768e;">%</span><span style="color: #89ddff;">;</span></div><div><span style="color: #9abdf5;"> </span><span style="color: #7aa2f7;">font-family</span><span style="color: #89ddff;">:</span><span style="color: #9abdf5;"> </span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">Noto Sans</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span><span style="color: #bb9af7;"> </span><span style="color: #9ece6a;">sans-serif</span><span style="color: #89ddff;">;</span></div><div><span style="color: #9abdf5;"> }</span></div><div><span style="color: #c0caf5;"> </span><span style="color: #ba3c97;"></</span><span style="color: #f7768e;">style</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"></</span><span style="color: #f7768e;">head</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">body</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">div</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">class</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;">pretext</span><span style="color: #89ddff;">"</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">div</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">style</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;"> display: flex; justify-content: center;</span><span style="color: #89ddff;">"</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">h2</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">style</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;">position:absolute; font-family: 'Noto Sans', sans-serif;</span><span style="color: #89ddff;">"</span><span style="color: #ba3c97;">></span><span style="color: #9aa5ce;"> [24]7 Synergen Embedded Dashboard </span><span style="color: #ba3c97;"></</span><span style="color: #f7768e;">h2</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"></</span><span style="color: #f7768e;">div</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">div</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">p</span><span style="color: #de5971;"> </span><span style="color: #bb9af7;">id</span><span style="color: #89ddff;">="</span><span style="color: #9ece6a;">dashboard-container</span><span style="color: #89ddff;">"</span><span style="color: #ba3c97;">></</span><span style="color: #f7768e;">p</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"></</span><span style="color: #f7768e;">div</span><span style="color: #ba3c97;">></span></div><div><div><span style="color: #89ddff;"> </span><span style="color: #ba3c97;"><</span><span style="color: #f7768e;">script</span><span style="color: #ba3c97;">></span></div><span style="color: #a9b1d6;"><br /></span><div><span style="color: #c0caf5;"> </span><span style="color: #9d7cd8; font-style: italic;">async</span><span style="color: #c0caf5;"> </span><span style="color: #bb9af7;">function</span><span style="color: #c0caf5;"> </span><span style="color: #7aa2f7;">fetchGuestTokenFromBackend</span><span style="color: #9abdf5;">()</span><span style="color: #c0caf5;"> </span><span style="color: #9abdf5;">{</span></div><div><span style="color: #89ddff;"> </span><span style="color: #444b6a; font-style: italic;">// let response = await fetch('https://mybackend.com/fetchGuestToken', { method: 'POST'});</span></div><div style="color: black;"><span style="color: #9abdf5;"> </span><span style="color: #9d7cd8; font-style: italic;">let</span><span style="color: #9abdf5;"> </span><span style="color: #bb9af7;">data</span><span style="color: #9abdf5;"> </span><span style="color: #89ddff;">=</span><span style="color: #9abdf5;"> </span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">guest_token_value_generated_using_above_python_code</span><span style="color: #89ddff;">'</span></div><div><span style="color: #9abdf5;"> </span><span style="color: #bb9af7; font-style: italic;">return</span><span style="color: #9abdf5;"> </span><span style="color: #c0caf5;">data</span><span style="color: #9abdf5;"> </span></div><div><span style="color: #9abdf5;"> }</span></div><div><span style="color: #c0caf5;"> supersetEmbeddedSdk</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">embedDashboard</span><span style="color: #9abdf5;">({</span></div><div><span style="color: #c0caf5;"> </span><span style="color: #73daca;">id</span><span style="color: #89ddff;">:</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">164104a5-69a8-484b-bb51-2a5cf7cb4a29</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #444b6a; font-style: italic;">// given by the Superset embedding UI</span></div><div><span style="color: #c0caf5;"> </span><span style="color: #73daca;">supersetDomain</span><span style="color: #89ddff;">:</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">https://superset.mydomain.com</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span></div><div><span style="color: #c0caf5;"> </span><span style="color: #73daca;">mountPoint</span><span style="color: #89ddff;">:</span><span style="color: #c0caf5;"> document</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">getElementById</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">dashboard-container</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #444b6a; font-style: italic;">// any html element that can contain an iframe</span></div><div><span style="color: #c0caf5;"> </span><span style="color: #7aa2f7;">fetchGuestToken</span><span style="color: #89ddff;">:</span><span style="color: #c0caf5;"> </span><span style="color: #9abdf5;">()</span><span style="color: #c0caf5;"> </span><span style="color: #bb9af7;">=></span><span style="color: #c0caf5;"> </span><span style="color: #7aa2f7;">fetchGuestTokenFromBackend</span><span style="color: #9abdf5;">()</span><span style="color: #89ddff;">,</span></div><div><span style="color: #c0caf5;"> </span><span style="color: #73daca;">dashboardUiConfig</span><span style="color: #89ddff;">:</span><span style="color: #c0caf5;"> </span><span style="color: #9abdf5;">{</span></div><div><span style="color: #c0caf5;"> </span><span style="color: #73daca;">hideTitle</span><span style="color: #89ddff;">:</span><span style="color: #ff9e64;">true</span><span style="color: #89ddff;">,</span></div><div><span style="color: #c0caf5;"> </span><span style="color: #73daca;">hideTab</span><span style="color: #89ddff;">:</span><span style="color: #ff9e64;">true</span><span style="color: #89ddff;">,</span></div><div><span style="color: #c0caf5;"> </span><span style="color: #73daca;">hideChartControl</span><span style="color: #89ddff;">:</span><span style="color: #ff9e64;">true</span></div><div><span style="color: #c0caf5;"> </span><span style="color: #9abdf5;">}</span><span style="color: #c0caf5;"> </span><span style="color: #444b6a; font-style: italic;">// dashboard UI config: hideTitle, hideTab, hideChartControls (optional)</span></div><div><span style="color: #c0caf5;"> </span><span style="color: #9abdf5;">})</span></div><div><span style="color: #9abdf5;"><br /></span></div><div><span style="color: #c0caf5;"> </span><span style="color: #ba3c97;"></</span><span style="color: #f7768e;">script</span><span style="color: #ba3c97;">></span></div></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"></</span><span style="color: #f7768e;">div</span><span style="color: #ba3c97;">></span></div><div><span style="color: #9aa5ce;"> </span><span style="color: #ba3c97;"></</span><span style="color: #f7768e;">body</span><span style="color: #ba3c97;">></span></div><div><span style="color: #ba3c97;"></</span><span style="color: #f7768e;">html</span><span style="color: #ba3c97;">>
</span></div><div><span style="color: #ba3c97;"><br /></span></div></div></span></div></span></div></span></div></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="text-align: left;">You can refer the <a href="https://github.com/apache/superset/tree/master/superset-embedded-sdk" rel="nofollow" target="_blank">superset embedded sdk </a>document for other configuration details.</span></p></div></div><div><br /></div><div><b>Troubleshooting</b></div><div><br /></div><div>If you encounter any challenges during the embedding process, consider the following configurations related to guest token and embedding feature:</div><div><br /></div></span><div><span style="font-family: Noto Sans;"><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6;"><span style="color: #bb9af7;"><div style="color: #a9b1d6;"><span style="color: #bb9af7;"><div style="color: #a9b1d6;"><span style="color: #89ddff;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6;"><div style="line-height: 20px;"><div><span style="color: #ff9e64;">FEATURE_FLAGS</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">EMBEDDED_SUPERSET</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">DASHBOARD_RBAC</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">ENABLE_TEMPLATE_PROCESSING</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">MENU_HIDE_USER_INFO</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">False</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">DRILL_TO_DETAIL</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">DASHBOARD_CROSS_FILTERS</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">True</span></div><div><span style="color: #89ddff;">}</span></div></div><span style="color: #ff9e64;">
SESSION_COOKIE_HTTPONLY</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">True</span> <span style="color: #444b6a; font-style: italic;"># Prevent cookie from being read by frontend JS?</span></div><div style="color: #a9b1d6;"><span style="color: #ff9e64;">SESSION_COOKIE_SECURE</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">True</span> <span style="color: #444b6a; font-style: italic;"># Prevent cookie from being transmitted over non-tls?</span></div><div style="color: #a9b1d6;"><span style="color: #ff9e64;">SESSION_COOKIE_SAMESITE</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">None</span><span style="color: #89ddff;">"</span> <span style="color: #444b6a; font-style: italic;"># One of [None, 'None', 'Lax', 'Strict']</span></div><div style="color: #a9b1d6;"><span style="color: #ff9e64;">SESSION_COOKIE_DOMAIN</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">False</span></div><span style="color: #a9b1d6;"><br /></span><div style="color: #a9b1d6;"><span style="color: #444b6a; font-style: italic;"># Cross-Origin</span></div><div style="color: #a9b1d6;"><span style="color: #ff9e64;">ENABLE_CORS</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">True</span> </div><div style="color: #a9b1d6;"><span style="color: #ff9e64;">CORS_OPTIONS</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">{</span> </div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">supports_credentials</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span> </div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">allow_headers</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">[</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">*</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">],</span> </div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">resources</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:[</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">*</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">],</span> </div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">origins</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">[</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">*</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">http://127.0.0.1:5500</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">],</span> </div><div style="color: #a9b1d6;"><span style="color: #89ddff;">}</span> </div><span style="color: #a9b1d6;"><br /></span><div style="color: #a9b1d6;"><span style="color: #444b6a; font-style: italic;"># Dashboard embedding</span></div><div style="color: #a9b1d6;"><span style="color: #ff9e64;">GUEST_ROLE_NAME</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Gamma</span><span style="color: #89ddff;">"</span></div><div style="color: black;"><span style="color: #ff9e64;">GUEST_TOKEN_JWT_SECRET</span><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">=</span><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">your_secret_key</span><span style="color: #89ddff;">"</span></div><div style="color: #a9b1d6;"><span style="color: #ff9e64;">GUEST_TOKEN_JWT_ALGO</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">HS256</span><span style="color: #89ddff;">"</span></div><div style="color: #a9b1d6;"><span style="color: #ff9e64;">GUEST_TOKEN_HEADER_NAME</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">X-GuestToken</span><span style="color: #89ddff;">"</span></div><div style="color: #a9b1d6;"><span style="color: #ff9e64;">GUEST_TOKEN_JWT_EXP_SECONDS</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">3600</span> <span style="color: #444b6a; font-style: italic;"># 60 minutes</span></div></span></div></span></div></span></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p></span><p style="text-align: left;"><span style="font-family: Noto Sans;">By following the outlined steps and utilizing the Superset Embedded SDK, you can seamlessly integrate Superset dashboards into your external applications. This empowers users with enhanced data visualization capabilities and enables efficient analysis within a unified environment. The combination of Superset's embedding feature and its robust data visualization capabilities makes it an invaluable tool for data-driven applications</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><h3><span style="font-family: Noto Sans;">Apache Superset Authentication: Implementing OAuth2 with Azure Identity Platform</span></h3><p><span style="font-family: Noto Sans;">Flask App Builder offers a range of authentication methods to bolster security within your applications. These include:</span></p><div><ul><li><span style="font-family: Noto Sans;">Database</span></li><li><span style="font-family: Noto Sans;">OpenID</span></li><li><span style="font-family: Noto Sans;">LDAP</span></li><li><span style="font-family: Noto Sans;">Remote User</span></li><li><span style="font-family: Noto Sans;">OAuth</span></li></ul><span style="font-family: Noto Sans;">Depending on your specific use case and requirements, Apache Superset enables you to select an authentication type from the supported options. To gain deeper insights into these authentication types and their implementation details, consult the<a href="https://flask-appbuilder.readthedocs.io/en/latest/security.html" rel="nofollow" target="_blank"> Flask App Builder Security documentation</a>.<br /><br />We will explore the process of implementing OAuth2 authentication using the Azure Identity Platform for Apache Superset.</span></div><div><span style="font-family: Noto Sans;"><br /></span></div><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Azure AD OAuth2 Authentication Implementation</span></h3><p style="text-align: left;"><span style="font-family: "Noto Sans";">Assuming you have registered your application in Azure AD and generated the required client secret for configuration purposes, follow these steps:</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";">Register your application in Azure AD and make note of the Tenant ID, Client ID, and Client Secret. If not done already, refer to the documentation on <a href="https://learn.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app" rel="nofollow" target="_blank">registering a client application with the Microsoft Identity Platform</a>.</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><b><br /></b></span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><b>Implement OAuth2 authentication for your Apache Superset instance:</b></span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";">Edit your Apache Superset configuration file (superset_config.py) and add the following configurations:</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6;"><span style="color: #bb9af7;"><div style="color: #a9b1d6;"><span style="color: #bb9af7;"><div style="color: #a9b1d6;"><span style="color: #89ddff;"><div style="color: #a9b1d6;"><div style="line-height: 20px;"><span style="color: #ff9e64;"><br class="Apple-interchange-newline" /><div style="color: #a9b1d6; line-height: 20px;"><span style="color: #7dcfff;">from</span> flask_appbuilder<span style="color: #89ddff;">.</span>security<span style="color: #89ddff;">.</span>manager <span style="color: #7dcfff;">import</span> <span style="color: #ff9e64;">AUTH_OAUTH</span></div><div style="color: #a9b1d6; line-height: 20px;"><span style="color: #ff9e64;"><br /></span></div><div style="color: #a9b1d6; line-height: 20px;"><div style="line-height: 20px;"><span style="color: #444b6a; font-style: italic;"># Set the authentication type to OAuth</span></div></div><div style="color: black; line-height: 20px;"><div style="color: #a9b1d6; line-height: 20px;"><span style="color: #ff9e64;">AUTH_TYPE</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">AUTH_OAUTH</span></div><div style="color: #a9b1d6; line-height: 20px;"><span style="color: #ff9e64;"><br /></span></div><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="color: #a9b1d6;"><span style="color: #444b6a; font-style: italic;"># Self registration & default role</span></div><div style="color: #a9b1d6;"><span style="color: #ff9e64;">AUTH_USER_REGISTRATION</span> <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">True</span></div><div><span style="color: #ff9e64;">AUTH_USER_REGISTRATION_ROLE</span><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">=</span><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Admin</span><span style="color: #89ddff;">"
</span><div style="line-height: 20px;"><div style="color: #a9b1d6;"><span style="color: #ff9e64;">OAUTH_PROVIDERS</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">[</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">{</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">name</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">azure</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">icon</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">fa-windows</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">token_key</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">access_token</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">remote_app</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">{</span></div><div><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">client_id</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">your client id</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">client_secret</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span><span style="color: #a9b1d6;"> </span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">your client secret</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">api_base_url</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">https://login.microsoftonline.com/tenant_id/oauth2</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">client_kwargs</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">{</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">scope</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">User.read name preferred_username email profile upn groups</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">resource</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">your client id</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">},</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">request_token_url</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #ff9e64;">None</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">access_token_url</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">https://login.microsoftonline.com/tenant_id/oauth2/token</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">authorize_url</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">https://login.microsoftonline.com/tenant_id/oauth2/authorize</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">},</span></div><div style="color: #a9b1d6;"> <span style="color: #89ddff;">},</span></div><div style="color: #a9b1d6;"><span style="color: #89ddff;">]</span></div></div></div></div></div></div></span></div></div></span></div></span></div></span></div></span></div><div style="line-height: 20px;"><span style="color: #c0caf5;"><br /></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: "Noto Sans"; text-align: start;">Create a custom Security Manager class that extends the Superset Security Manager class with the following settings:</span></p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: "Noto Sans"; text-align: start;"><br /></span></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #c0caf5;"><div style="color: #a9b1d6;"><span style="color: #bb9af7;"><div style="color: #a9b1d6;"><span style="color: #bb9af7;"><div style="color: #a9b1d6;"><span style="color: #89ddff;"><div style="color: #a9b1d6;"><div style="line-height: 20px;"><span style="color: #ff9e64;"><div style="color: #a9b1d6; line-height: 20px;"><div><span style="color: #7dcfff;">
import</span> logging</div><div><span style="color: #7dcfff;">from</span> superset<span style="color: #89ddff;">.</span>security <span style="color: #7dcfff;">import</span> SupersetSecurityManager</div><br /><div><span style="color: #bb9af7;">class</span> <span style="color: #c0caf5;">CustomSsoSecurityManager</span><span style="color: #89ddff;">(</span><span style="color: #bb9af7;">SupersetSecurityManager</span><span style="color: #89ddff;">)</span><span style="color: #9abdf5;">:</span></div><div> <span style="color: #bb9af7;">def</span> <span style="color: #7aa2f7;">_get_oauth_user_info</span><span style="color: #9abdf5;">(</span><span style="color: #e0af68;">self</span><span style="color: #89ddff;">,</span> <span style="color: #e0af68;">provider</span><span style="color: #89ddff;">,</span> <span style="color: #e0af68;">resp</span><span style="color: #89ddff;">=</span><span style="color: #ff9e64;">None</span><span style="color: #9abdf5;">):</span></div><div> <span style="color: #444b6a; font-style: italic;">#logging.debug("Oauth2 provider: {0}.".format(provider))</span></div><br /><div> <span style="color: #bb9af7;">if</span> provider <span style="color: #bb9af7;">==</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">azure</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span></div><div> <span style="color: #444b6a; font-style: italic;">#logging.debug("Azure response received : {0}".format(resp))</span></div><div> id_token <span style="color: #89ddff;">=</span> resp<span style="color: #9abdf5;">[</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">id_token</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">]</span></div><div> <span style="color: #444b6a; font-style: italic;">#logging.debug(str(id_token))</span></div><div> me <span style="color: #89ddff;">=</span> <span style="color: #f7768e;">self</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">_azure_jwt_token_parse</span><span style="color: #9abdf5;">(</span><span style="color: #c0caf5;">id_token</span><span style="color: #9abdf5;">)</span></div><div> <span style="color: #444b6a; font-style: italic;">#logging.debug("Parse JWT token : {0}".format(me))</span></div><div> <span style="color: #bb9af7;">return</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">name</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> me<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">get</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">name</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">""</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">email</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> me<span style="color: #9abdf5;">[</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">upn</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">]</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">first_name</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> me<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">get</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">given_name</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">""</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">last_name</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> me<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">get</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">family_name</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">""</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">id</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> me<span style="color: #9abdf5;">[</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">oid</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">]</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">username</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> me<span style="color: #9abdf5;">[</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">oid</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">]</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">role_keys</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> me<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">get</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">roles</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #c0caf5;"> </span><span style="color: #89ddff;">[]</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">}</span></div><br /><div> oauth_user_info <span style="color: #89ddff;">=</span> _get_oauth_user_info
</div></div></span></div></div></span></div></span></div></span></div></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><br /></p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: "Noto Sans"; text-align: start;">Save this file as custom_sso_security_manager.py and import the Custom Security Manager in your configuration file:</span></p><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: "Noto Sans"; text-align: start;"><br /></span></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><span style="color: #7dcfff;">
from</span> custom_sso_security_manager <span style="color: #7dcfff;">import</span> CustomSsoSecurityManager
<div style="line-height: 20px;"><span style="color: #ff9e64;">CUSTOM_SECURITY_MANAGER</span> <span style="color: #89ddff;">=</span> CustomSsoSecurityManager
</div><div><br /></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p class="p2" style="font-feature-settings: normal; font-kerning: auto; font-optical-sizing: auto; font-stretch: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variation-settings: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: "Noto Sans"; text-align: start;"><br />Update your client application's redirect URL:<br /><br /></span></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; font-family: "Roboto Mono", monospace; font-size: 13px; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="background-color: #1a1b26; color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;"><div style="line-height: 20px;">
</div><div style="line-height: 20px;"><div style="line-height: 20px;">https<span style="color: #89ddff;">://</span>superset<span style="color: #89ddff;">.</span>mydomain<span style="color: #89ddff;">.</span>com<span style="color: #89ddff;">/</span>oauth<span style="color: #89ddff;">-</span>authorized<span style="color: #89ddff;">/</span>azure</div></div><div><br /></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></pre></div><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br />After restarting, you will notice the updated login page reflecting your changes.<br /><br /></span><span style="font-family: Noto Sans;">Azure AD:<br /></span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnFBeBYAOrov6M06zqi2DGQLuS_E9jlJfhM0WFTu8qzl2ChfzEyY-TlS1YYNpY-MMQYfkJAP8xyx8aamiD_qoC4kNZUfqxEbX8O74l1WIbTDgV32pF13O5WmutJMp5wFt3BvKFgJ2gufm1Uc1iDOIUc-pxseS2xNStNOoc37w9Vqx3_hmIICRYvtn7ju9m/s1234/oauth.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="Apache Superset OAuth2- Azure AD Authentication" border="0" data-original-height="396" data-original-width="1234" height="206" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnFBeBYAOrov6M06zqi2DGQLuS_E9jlJfhM0WFTu8qzl2ChfzEyY-TlS1YYNpY-MMQYfkJAP8xyx8aamiD_qoC4kNZUfqxEbX8O74l1WIbTDgV32pF13O5WmutJMp5wFt3BvKFgJ2gufm1Uc1iDOIUc-pxseS2xNStNOoc37w9Vqx3_hmIICRYvtn7ju9m/w640-h206/oauth.png" title="Apache Superset OAuth2- Azure AD Authentication" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Apache Superset OAuth2- Azure AD Authentication<br /><br /></td></tr></tbody></table><p><span style="font-family: Noto Sans;"><br /></span></p><p><span style="font-family: Noto Sans;">If you have configured multiple Identity Providers, you can see multiple providers option in sign-in page:</span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgTqLgquJloGaLxY3eWKMXKDAdaN7gqQCrcU6emX2oVLHGdfRWYu5Z9LB_QRuMzYY5-wvcDvRz07ytB1z2g0RIGgSpg6i5zOTIlXzzXyA3Mp6IC7snGdAHWHxzUkMl-2TDKQnTw7gpxz89imB6iAEcZUJW70MD4HMllTxOxAIZw1vgUOyehvuE-dA3J3vhw" style="margin-left: auto; margin-right: auto;"><img alt="Apache Superset OAuth2 Authentication" data-original-height="754" data-original-width="2344" height="206" src="https://blogger.googleusercontent.com/img/a/AVvXsEgTqLgquJloGaLxY3eWKMXKDAdaN7gqQCrcU6emX2oVLHGdfRWYu5Z9LB_QRuMzYY5-wvcDvRz07ytB1z2g0RIGgSpg6i5zOTIlXzzXyA3Mp6IC7snGdAHWHxzUkMl-2TDKQnTw7gpxz89imB6iAEcZUJW70MD4HMllTxOxAIZw1vgUOyehvuE-dA3J3vhw=w640-h206" title="Apache Superset OAuth2 Authentication" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Apache Superset OAuth2 Authentication</td></tr></tbody></table><p><span style="font-family: Noto Sans;"><br /></span></p><p><span style="font-family: "Noto Sans";">By following these steps, you can successfully implement OAuth2 authentication with the Azure Identity Platform for your Apache Superset instance. This integration ensures a secure authentication process while enabling seamless user access to the platform.</span></p><p><span style="font-family: Noto Sans;"><br /></span></p></div></div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><div style="text-align: left;"><span style="font-family: Noto Sans;"><h2 style="text-align: justify;">Benefits and Features of Apache Superset</h2><div style="text-align: justify;"><br /></div><div style="text-align: justify;">Apache Superset offers a wide range of benefits and features that make it a powerful tool for business intelligence and data visualisation:</div><div style="text-align: justify;"><b><br /></b></div></span></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><b>User-Friendly Interface:</b> Superset provides an intuitive and user-friendly interface that makes it easy for users to explore and analyse data. With drag-and-drop functionality and interactive visualisations, users can create insightful dashboards without writing complex queries or code.</div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><b>Wide Range of Data Sources:</b> Superset supports a variety of data sources, including popular databases like MySQL, PostgreSQL, and SQLite, as well as big data platforms like Apache Hive, Apache Spark, and Presto. It also integrates with cloud-based storage solutions like Amazon S3 and Google Cloud Storage.</div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><b>Interactive Dashboards:</b> Superset allows users to create interactive dashboards with a wide range of visualization options, including charts, graphs, maps, and tables. Users can customize the appearance and layout of dashboards and easily share them with others.</div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><b>Ad-Hoc Analysis:</b> With Superset, users can perform ad-hoc analysis by exploring and filtering data in real-time. The SQL Lab feature allows users to write and execute SQL queries directly in the browser, providing instant results and insights.</div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><b>Collaboration and Sharing:</b> Superset enables collaboration by allowing users to share dashboards, charts, and SQL queries with others. Users can set permissions and access controls to ensure data security and privacy.</div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><b>Extensibility and Customization:</b> Superset is highly extensible and customizable, allowing users to add custom visualizations, plugins, and integrations. The Superset community actively contributes to the development of new features and enhancements.</div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><b>Scalability and Performance:</b> Superset is designed to scale out in distributed environments and can handle large datasets and high user concurrency. It leverages technologies like Apache Druid and Apache Arrow to provide fast and efficient data processing.</div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div></span></div><div><span style="font-family: Noto Sans;"><div style="text-align: justify;"><b>Active Community Support:</b> Apache Superset has a vibrant and active community of users and contributors. The community provides support, documentation, and regular updates, ensuring that Superset remains a robust and reliable tool for business intelligence.</div></span></div></blockquote><div style="text-align: left;"><span style="font-family: Noto Sans;"><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><br /></div></span></div></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">In this article, we learned how to install Apache Superset on a Linux machine using pip and how to initialize the Superset database, create an admin user, load some example data sets, start the Superset server, and access the web interface. We also saw some screenshots of what Superset looks like and what it can do.</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">Apache Superset is a powerful and easy-to-use data visualization and exploration platform that can help you gain insights from your data and communicate them effectively. You can learn more about Superset by visiting its official website, documentation, and GitHub repository.</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">I hope you enjoyed this article and found it useful. Happy data visualization!</div></span></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com0tag:blogger.com,1999:blog-2027709621250502166.post-61644437056644982842023-03-09T15:30:00.003+05:302023-03-09T20:27:16.269+05:30GCP Cloud Pub/Sub Replay: Seeking to timestamp & Seeking to snapshots<p style="text-align: left;"><span style="font-family: Noto Sans;"></span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVk8xHuFb0Ss6vsszUtS9ngFUCNm_c4hw18yE8smKM6cByoCYo8apXWWRhJVoqCqz4FQ1LD8r9EwxQLTJA-2yQvC6G5q3UbTktmsoHTuGGx53GP6IiZQ-u4PYmnpFWExrLU50KJvlJh1q234KU-Pp5xpQ2CmFptYUky9l-INsSoFAqofVfglw4DEIYtw/s1920/loop-button-5484820.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="Google Cloud Pub/Sub Replay (Pixabay)" border="0" data-original-height="1920" data-original-width="1920" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVk8xHuFb0Ss6vsszUtS9ngFUCNm_c4hw18yE8smKM6cByoCYo8apXWWRhJVoqCqz4FQ1LD8r9EwxQLTJA-2yQvC6G5q3UbTktmsoHTuGGx53GP6IiZQ-u4PYmnpFWExrLU50KJvlJh1q234KU-Pp5xpQ2CmFptYUky9l-INsSoFAqofVfglw4DEIYtw/w400-h400/loop-button-5484820.png" title="Google Cloud Pub/Sub Replay (Pixabay)" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Google Cloud Pub/Sub Replay (Pixabay) </span></td></tr></tbody></table><span style="font-family: Noto Sans;"><br style="text-align: left;" /></span></td></tr></tbody></table><span style="font-family: Noto Sans;"><br /></span><p></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Let's assume, you have data pipeline deployed on Google Cloud Platform, events are published to Cloud Pub/Sub topic from publisher client, and subscribed by a data processing application, which reads data from the Cloud Pub/Sub subscription, process it </span><span style="font-family: "Noto Sans";">and write it to BigQuery table.</span></p><p style="text-align: left;"><span></span></p><a name='more'></a><span style="font-family: "Noto Sans";"><br /></span><p></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Consider a scenario, due to a in the publisher client, the published events are corrupted and not processed by the data processing subscriber and crashing the data processing application. Also, you are observing the increase in the unacknowledged message count and event backlog size metrics. After a day you have fixed the publisher code and redeployed the publisher client. But still you are seeing the subscriber client is crashing and metrics are continue to go up, this is due to the corrupted events already published to Cloud Pub/Sub subscription and waiting for subscriber client to acknowledge. So how do we handle those unacknowledged messages?</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Before discussing solution to above problem, let's consider another scenario, you have made some changes to data processing logic and redeployed the subscriber code. Now you are seeing the metrics are okay, not seeing any increase in unacknowledged message counts and backlog size, but data is not available in BigQuery table. So what is happening here, due to a bug in the subscriber code, the messages are being acknowledged by the data processing application but not actually processed and written to BigQuery table, therefore data loss. So how do we retrieve those lost messages?</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">In the former scenario, there are unacknowledged corrupted events, we don't want to process and skip those events and need to fast-forward/purge events until we get to the time period where we started receiving valid events (events published after the bug fixed in the publisher client). In the later scenario, we have already wrongly acknowledged events (though not processed by subscriber code due to the bugs), now we need to rewind and un-acknowledge those events process them again (we have to go back to the time period when the subscriber code was redeployed, un-acknowledge and replay those events).</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Cloud Pub/Sub Seek:</span></h2><span style="font-family: Noto Sans;">In regular setup you can retrieve the acknowledged messages from Cloud Pub/Sub. However, as we have seen above, sometimes we need to retrieve the previously acknowledged messages to reprocess them (bulk un-acknowledge - change the message status to unacknowledged at once) or fast-forward to purge the unacknowledged messages (bulk acknowledge - change the message status to acknowledged at once).<br /><br />GCP Cloud Pub/Sub provides Seek API to seek to a specific timestamp or seek to a snapshot to replay the message in the subscription.<br /><br /></span><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Seek to a timestamp:</span></h2><span style="font-family: Noto Sans;">To seek to a timestamp in the past and unacknowledge the already acknowledged messages in bulk, you need to retain the acknowledged messages in Cloud Pub/Sub topic or subscription. Let's discuss about Cloud Pub/Sub message retention below:</span><div><span style="font-family: Noto Sans;"><br /></span></div><div><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Cloud Pub/Sub topic message retention:</span></h3></div><div><span style="font-family: Noto Sans;">When topic retention is enabled, Cloud Pub/Sub topic is responsible for storing acknowledged messages and topic has the full control over the message retention duration, independent of subscriber retention settings. You can enable the message retention at the topic as follows:<br /></span></div><div><span style="font-family: Noto Sans;"><br /></span></div><div><span style="font-family: Noto Sans;">Create a Cloud Pub/Sub topic with message retention duration of 2 weeks:</span></div><div><span style="font-family: Noto Sans;"><br /></span></div><div><div style="background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; text-rendering: optimizelegibility; vertical-align: middle; white-space: pre-wrap; width: 1124px; word-break: normal;"><span style="-webkit-font-smoothing: antialiased; background-color: #171717; border: 0px; box-sizing: border-box; color: #e6e6e6; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-rendering: optimizelegibility; vertical-align: baseline; white-space: normal;"><span style="-webkit-font-smoothing: antialiased; border: 0px; box-sizing: border-box; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; text-rendering: optimizelegibility; vertical-align: baseline;"><span style="font-family: Source Code Pro;">gcloud pubsub topics create seek-demo-topic --message-retention-duration=14d</span></span></span></div><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">To enable message retention on existing topic:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub topics update seek-demo-topic --message-retention-duration=1d</span></span></pre><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">To disable message retention on a topic:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;"></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub topics update seek-demo-topic --clear-message-retention-duration</span></span></pre><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;"><br /></span></p><h3 style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;"><span style="font-size: medium;">Cloud Pub/Sub subscription message retention:</span></span></h3><p style="text-align: left;"><span style="font-family: Noto Sans;"><span>Pub/Sub discards a message from subscription as soon as the message is acknowledged. Unacknowledged messages are retained in subscription for 7 days. To replay acknowledged messages from Pub/Sub subscription, you need to retain the acknowledged messages at the subscription. When topic retention is set greater than 7 days, acknowledged messages are retained more than 7 days (maximum of topic & subscription retention period). </span></span></p><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;"><span style="font-size: medium;"><br /></span></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><span>To create subscription with acknowledged message retention:</span></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><span><br /></span></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub subscriptions create seek-demo-subscription --topic=seek-demo-topic --retain-acked-messages --message-retention-duration=6d</span></span></pre><p style="text-align: left;"><span style="font-family: Noto Sans; font-size: medium;"><span style="font-size: medium;"><br /></span></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">To update existing subscription's acknowledged message retention:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub subscriptions update seek-demo-subscription --message-retention-duration=1d</span></span></pre><p style="text-align: left;"><br /></p><h3 style="text-align: left;"><span style="font-family: "Noto Sans";">Acknowledged message retention: topic vs subscription:</span></h3><p style="text-align: left;"><span style="font-family: "Noto Sans";">Cloud Pub/Sub provides message retention options both at topic and subscription level. Choosing the right retention option is important as it impact the cost and maintenance as well. Lets discuss about important considerations in choosing the right retention option for you.</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><h4 style="text-align: left;"><span style="font-family: "Noto Sans";"></span></h4><blockquote><h4 style="text-align: left;"><span style="font-family: "Noto Sans";">Topic retention: </span></h4><p><span style="font-family: "Noto Sans";">Configured at topic level - maintained by topic owner at topic project, pay once for all attached subscription (including the other projects subscriptions). </span><span style="font-family: Noto Sans;">Messages will be retained and available for replay even if there are no subscriptions attached to the topic at the time the messages are published. </span></p><h4 style="text-align: left;"><span style="font-family: Noto Sans;">Subscription retention:</span></h4><p style="text-align: left;"><span style="font-family: "Noto Sans";">Configured individually at the subscription level at each subscription projects by respective subscription owners. Pay separately for the subscription retention storage at the subscription project. Comparatively higher maintenance than topic retention, but gives more flexible to retain only the required subset of the messages. </span></p><p style="text-align: left;"></p></blockquote><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><h2 style="text-align: left;"><span style="font-family: "Noto Sans";">Seeking to a timestamp:</span></h2><p style="text-align: left;"><span style="font-family: "Noto Sans";">In our first scenario, we have seen that there are unacknowledged corrupted events, processing which will crash the subscriber code, and we discussed the possible solution of purging those events / fast-forwarding to valid events will solve the issue. Seeking to a timestamp helps to achieve this solution.</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";">For example, let say, your publisher client publishing around 100 events per seconds and there were some changes at the publisher client at 07th March, 2023 - 05:00 PM and all the events published to Cloud Pub/Sub after this time crashing the subscriber code. You have identified the issue and implemented fix and redeployed the publisher code by 08th, Mar 2023 - 09:00 AM. But still the subscriber is getting crashed due to the unacknowledged events, that is events published between 2023-03-07T17:00:00 to 2023-03-08T09:00:00. </span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";">We have flexibility to skip those events, so we decided to purge those events, that is changing the events status from unacknowledged to acknowledged, therefore, our subscriber will receives</span><span style="font-family: "Noto Sans";"> only valid messages.</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">At 2023-03-08T09:00:00 you have updated the new publisher client code and seek to this time (current datetime) to purge all unacknowledged events. That is any backlogs before </span><span style="font-family: "Noto Sans";">2023-03-08T09:00:00 will be purged / event's status changed to acknowledged. Command for seeking to a current timestamp (</span><span style="font-family: "Noto Sans";">2023-03-08T09:00:00)</span><span style="font-family: "Noto Sans";"> given below:</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">export TS_FORMAT=%Y-%m-%dT%H:%M:%SZ<br />gcloud pubsub subscriptions seek seek-demo-subscription --time=$(date -u +$TS_FORMAT)</span></span></pre><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Also, you can specify the timestamp explicitly to a required time as follows:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub subscriptions seek seek-demo-subscription --time=2023-03-08T09:00:00.000000Z</span></span></pre><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Therefore, now the subscriber will receive only events published after </span><span style="font-family: "Noto Sans";">2023-03-08T09:00:00. That is subscriber's backlog reset to current point in time. Please remember, seeking to a timestamp will work, only if the message retention is enabled at the topic or subscriber.</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><h2 style="text-align: left;"><span style="font-family: "Noto Sans";">Seeking to a snapshot:</span></h2><p style="text-align: left;"><span style="font-family: Noto Sans;">In our second scenario we have seen that events are already acknowledged by the subscriber but not successfully processed due to a bug in the newly deployed code. Also, we have discussed about the possible solution of rewind/replaying the acknowledged events to reprocess, that is change the event's status from acknowledged to unacknowledged, so that Cloud Pub/Sub will be deliver those messages again.</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">We have seen how to rewind / fast-forward using seeking to a timestamp functionality above. In this section, we will see how to replay events using seeking to a snapshot functionality.</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Let say you have made some changes to data processing logic and redeploy the subscriber code at 07th March, 2023 05:00 PM. And you observed that, due to a bug in the code events are acknowledged but not processed and written to BigQuery table. Therefore you want to fix the subscriber code and seek to 2023-03-07T17:00:00, that is unacknowledge all the acknowledged events from </span><span style="font-family: "Noto Sans";">2023-03-07T17:00:00. </span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";">To seek to a snapshot, first you need to create a snapshot, as we are making changes to subscriber code, create a snapshot just before the deployment (i.e </span><span style="font-family: "Noto Sans";">2023-03-07T17:00:00) as below:</span></p><p style="text-align: left;"><br /></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub snapshots create seek-demo-snapshot --subscription=seek-demo-subscription</span></span></pre><p style="text-align: left;"><br /></p><p style="text-align: left;"><span style="font-family: "Noto Sans";">You can list the created snapshots and expiration time by running below command:</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub snapshots list<br /></span></span></pre><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"></span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhOty2rlYQAQrikQ6Klqwy1U-m4QrG-T0p65IbkqkM9lSv4uaqeSfCIIJHgGisLy9py8IcT3_0D2kOhzXa6DV6iuHb6Y8dETzdUzJ1xsgRixW9J7cbta1OYuX4v1WGRbgdqY26IW4876fkg7o-LwoF1q0xTmj1zZibQ-3nluYGoe2IvoJnA2Mktj07S4A" style="margin-left: auto; margin-right: auto;"><img alt="Cloud Pub/Sub List" data-original-height="424" data-original-width="2184" height="124" src="https://blogger.googleusercontent.com/img/a/AVvXsEhOty2rlYQAQrikQ6Klqwy1U-m4QrG-T0p65IbkqkM9lSv4uaqeSfCIIJHgGisLy9py8IcT3_0D2kOhzXa6DV6iuHb6Y8dETzdUzJ1xsgRixW9J7cbta1OYuX4v1WGRbgdqY26IW4876fkg7o-LwoF1q0xTmj1zZibQ-3nluYGoe2IvoJnA2Mktj07S4A=w640-h124" title="Cloud Pub/Sub List" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Noto Sans; font-size: x-small;">Cloud Pub/Sub List</span></td></tr></tbody></table><br /><br /><p></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";">Once a snapshot is created, it retains backlogs (all unacknowledged events in source subscription at the time of snapshot creation) and any events published to the topic thereafter.</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";">Now we have a snapshot created at the time of subscriber code deployment, if there is erroneous acknowledgement at the subscriber code, we can seek back to the time period using snapshot as below.</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub subscriptions seek seek-demo-subscription --snapshot=seek-demo-snapshot</span></span></pre><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";">Now you will be able to replay the acknowledged events from the snapshot. Seek operations works based on eventual consistency, so you may not see those events instantly after the seek operation, as per documentations, it may take a minute to deliver those events.</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Also, you can seek to snapshot from any of the attached subscription to the same topic or newly created subscriptions to the same topic, this will be helpful, if you would like to process those events separately. Below steps shows seek from different snapshot or newly created subscription:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Create new subscription:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub subscriptions create seek-demo-sub2 --topic=seek-demo-topic --ack-deadline=10</span></span></pre><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Seek to existing snapshot from newly created subscription:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub subscriptions seek seek-demo-sub2 --snapshot=seek-demo-snapshot</span></span></pre><div><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;"><br /></span></span></div><p style="text-align: left;"><span style="font-family: Noto Sans;">Snapshots are expired and deleted in 7 seven days or when it reaches the oldest unacknowledged event in the snapshot. For example, if the age of oldest unacknowledged event is 5 days, then the snapshot will expire in 2 days.</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Seeking events with Filters:</span></h3><p style="text-align: left;"><span style="font-family: Noto Sans;">Cloud Pub/Sub supports replay events from subscription with filters. When you use filters, Cloud Pub/Sub redelivers only events which matches the filter condition. Please note that, when you create a snapshot, it includes all the events, not just the events which matches the filtered condition, therefore storage cost applied to all the events, not just filtered event subset.</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Let's publish few more events to our demo topic and demonstrate the seeking with filters</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub topics publish seek-demo-topic --message '{"""Country""": """India""", """Capital""": """Beijing"""}' --attribute=Country="India"</span></span></pre><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub topics publish seek-demo-topic --message '{"""Country""": """China""", """Capital""": """Moscow"""}' --attribute=Country="China"</span></span></pre><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub topics publish seek-demo-topic --message '{"""Country""": """Russia""", """Capital""": """New Delhi"""}' --attribute=Country="Russia"</span></span></pre><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">As we already created our demo snapshot, which includes these messages as well. Let's create a subscription with message filter and seek filtered events from the snapshot.</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Creating new subscription with message filtering:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub subscriptions create seek-demo-india --topic=seek-demo-topic --ack-deadline=10 --message-filter='attributes.Country="India"'</span></span></pre><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Create one more subscription with different message filter:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub subscriptions create seek-demo-russia --topic=seek-demo-topic --ack-deadline=10 --message-filter='attributes.Country="Russia"'</span></span></pre><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Seeking to snapshot from seek-demo-india subscription:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub subscriptions seek seek-demo-india --snapshot=seek-demo-snapshot</span></span></pre><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";">Seeking to snapshot from seek-demo-russia subscription:</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub subscriptions seek seek-demo-russia --snapshot=seek-demo-snapshot</span></span></pre><div><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;"><br /></span></span></div><p style="text-align: left;"><span style="font-family: "Noto Sans";">Pull messages from seek-demo-india:</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub subscriptions pull seek-demo-india</span></span></pre><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Result:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"></span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjPX3jWtvzJq0KeUvfO-T19n8Y8FmEwhQTtK6MM8IRX-1BJQ94aA5D-p86hNZqrMDBV4Er-AHOtlhG3gvzlbi1GqamK4RBsMUifEluEjsGrIdjAZCAiCK2PuADN5agjuv8sdfNzHa1OtKZ-pEvV0bq389AJ6ZqeW4ANQ8m8vitgoOCYuM-dPQQBFgPk1A" style="margin-left: auto; margin-right: auto;"><img alt="Seeking to a snapshot with message filter" data-original-height="526" data-original-width="3584" height="94" src="https://blogger.googleusercontent.com/img/a/AVvXsEjPX3jWtvzJq0KeUvfO-T19n8Y8FmEwhQTtK6MM8IRX-1BJQ94aA5D-p86hNZqrMDBV4Er-AHOtlhG3gvzlbi1GqamK4RBsMUifEluEjsGrIdjAZCAiCK2PuADN5agjuv8sdfNzHa1OtKZ-pEvV0bq389AJ6ZqeW4ANQ8m8vitgoOCYuM-dPQQBFgPk1A=w640-h94" title="Seeking to a snapshot with message filter" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Seeking to a snapshot with message filter</span></td></tr></tbody></table><span style="font-family: Noto Sans;"><br /><br /></span><p></p><p style="text-align: left;"><span style="font-family: "Noto Sans";">Pull messages from seek-demo-russia:</span></p><p style="text-align: left;"><span style="font-family: "Noto Sans";"><br /></span></p><pre class="highlight" style="-webkit-font-smoothing: antialiased; background: rgb(10, 18, 27); border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; height: auto; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; outline: none; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-rendering: optimizelegibility; vertical-align: middle; width: 1124px; word-break: normal;"><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;">gcloud pubsub subscriptions pull seek-demo-russia</span></span></pre><div><span style="color: #e6e6e6; font-family: Source Code Pro;"><span style="white-space: normal;"><br /></span></span></div><p style="text-align: left;"><span style="font-family: Noto Sans;">Result:</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"></span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiVXLxW_3ZHzboYqXmLPRsqRc3uTNJ8YdbttPfo0uzX2VaYqiXFhY3BimVfRcvu8tZF9I7XXHpGu2QuV-3W3DaPpiV39ulTeaP22N0axn5kjouez6MkgYpJr39mxKOrwktVBmeVspKiO29_1fQqNxGE5tyPGe7px4z7Y6xarpS1ZV-LvdmLl41vVaIEIg" style="margin-left: auto; margin-right: auto;"><img alt="Seeking to a snapshot with message filter" data-original-height="1174" data-original-width="3584" height="210" src="https://blogger.googleusercontent.com/img/a/AVvXsEiVXLxW_3ZHzboYqXmLPRsqRc3uTNJ8YdbttPfo0uzX2VaYqiXFhY3BimVfRcvu8tZF9I7XXHpGu2QuV-3W3DaPpiV39ulTeaP22N0axn5kjouez6MkgYpJr39mxKOrwktVBmeVspKiO29_1fQqNxGE5tyPGe7px4z7Y6xarpS1ZV-LvdmLl41vVaIEIg=w640-h210" title="Seeking to a snapshot with message filter" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: small;">Seeking to a snapshot with message filter<br /></span></td></tr></tbody></table><span style="font-family: Noto Sans;"><br /><br /></span><p></p><p style="text-align: left;"><span style="font-family: Noto Sans;">Note, as of today (Mar, 2023) Cloud Pub/Sub allows you to filter events by attributes, not by message payload. Therefore, use event attribute to filter the events.</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Cloud Pub/Sub Replay - Use-cases:</span></h2><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><p style="text-align: left;"><span style="font-family: Noto Sans;">We have discussed two scenarios above where Pub/Sub replay functionality is very helpful. You can use the Cloud Pub/Sub Replay functionality, when deploying new publisher client code, subscriber client code, recovering from unexpected subscriber client code, testing subscriber code on real data, and all the other scenarios, where you need either acknowledge messages in bulk (fast-foreward) or unacknowledge previously acknowledged messages (rewind). Also, as Cloud Pub/Sub seek functionality allows you to perform operation in bulk of messages, it will be huge cost and time saver.</span></p><p style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></p><h2 style="text-align: left;"><span style="font-family: Noto Sans;">Summary:</span></h2><p style="text-align: left;"><span style="font-family: Noto Sans;">We have discussed about the scenarios where you need to acknowledge the unsupported messages in bulk which are waiting for delivery and the scenario where you need to redeliver the messages for reprocessing which already erroneously acknowledged by the subscriber code. And we have seen how these scenarios can be handled by using Cloud Pub/Sub Seek functionality where you forward or rewind to a point in time or to a snapshot. Also, we have seen the implementation details of enabling message retention, seeking to a timestamp and seeking to a snapshot. Finally, we have seen the seeking events with filter option and its implementation and use cases of Cloud Pub/Sub Seek functionality. I hope this article is helpful and please share your thoughts in comment section. Thank you!</span></p></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com0tag:blogger.com,1999:blog-2027709621250502166.post-63694970261301667322023-03-02T21:20:00.001+05:302023-03-02T21:20:56.043+05:30[Solved] Access is denied - Check credentials and try again - Microsoft Graph - Calendar API<p><span style="font-family: Noto Sans;"></span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg1AE8e-kSUuJSbRxdzfvLXMXFF7X88FQVFfEnfvb91uUon-8IS4iB5XrBfbxlcJhEkG4RuYLKTiJF5R8NCuXU7WQOWubC1LPOSak1bAkANn0YFnSLE3S6EzpdOwpv0ysVOvbqwLSJ6wavglwEcfnImBcg90FxQh9Pav3QAQNedo6wq6sPa9qpfMNez7w" style="margin-left: auto; margin-right: auto;"><img alt="Access is denied - Check credentials and try again - Microsoft Graph - Calendar API" data-original-height="391" data-original-width="800" src="https://blogger.googleusercontent.com/img/a/AVvXsEg1AE8e-kSUuJSbRxdzfvLXMXFF7X88FQVFfEnfvb91uUon-8IS4iB5XrBfbxlcJhEkG4RuYLKTiJF5R8NCuXU7WQOWubC1LPOSak1bAkANn0YFnSLE3S6EzpdOwpv0ysVOvbqwLSJ6wavglwEcfnImBcg90FxQh9Pav3QAQNedo6wq6sPa9qpfMNez7w=s16000" title="Access is denied - Check credentials and try again - Microsoft Graph - Calendar API" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: xx-small;">Microsoft Graph (Source: microsoft.com)</span></td></tr></tbody></table><span style="font-family: Noto Sans;"><br /><br /></span><p></p><p><span style="font-family: Noto Sans;">When sending API request to Microsoft Graph API, it responds with access denied error. You might have followed the documentation and added the correct permission and granted admin consent for the same, but it still produces the same error. Lets check the solution for this issue in this short article.</span></p><h3 style="text-align: left;"><span><a name='more'></a></span><span style="font-family: Noto Sans;"><br /></span></h3><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Error Message:</span></h3><div><span style="font-family: Noto Sans;"><br /></span></div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="font-family: Source Code Pro;">{'error': {'code': 'ErrorAccessDenied', 'message': 'Access is denied. Check credentials and try again.'}}</span></pre><h3 style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></h3><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Error Scenario:</span></h3><p><span style="font-family: Noto Sans;">This issue occurs while sending GET request to Microsoft Graph CalendarView resource.</span></p><p><span style="font-family: Noto Sans;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="white-space: normal;"><span style="color: white; font-family: Source Code Pro;">GET /users/{id | userPrincipalName}/calendars/{id}/calendarView?startDateTime={start_datetime}&endDateTime={end_datetime}</span></span></pre><h3 style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></h3><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Possible Reason:</span></h3><p><span style="font-family: Noto Sans;">As the error message clearly suggests, in most scenarios - Microsoft Graph API responds with this error message when there is a permission issue.</span></p><h3 style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></h3><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Permissions Required:</span></h3><p><span style="font-family: Noto Sans;">For reading users calendar from application requires following permissions with type of Application Permission. </span></p><p><span style="font-family: Noto Sans;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="background-color: #171717; color: #e6e6e6; white-space: normal;"><span style="font-family: Source Code Pro;">Calendars.Read OR Calendars.ReadWrite</span></span></pre><h3 style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></h3><h3 style="text-align: left;"><span style="font-family: Noto Sans;">Workaround:</span></h3><p><span style="font-family: Noto Sans;">This issue may occur due to many reasons. You could try one of the following solutions to fix this issue.</span></p><h4 style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></h4><h4 style="text-align: left;"><span style="font-family: Noto Sans;">Verify and remove the excessive permissions:</span></h4><p><span style="font-family: Noto Sans;">Check the least required access for the given request and remove all other access. In this case for accessing calendarview endpoint - Calendars.Read permission is sufficient remove all other permissions including the <span style="background-color: #404245; color: #e7e9eb; font-size: 13px; white-space: pre-wrap;">Calendars.ReadBasic.All</span> permission.</span></p><h4 style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></h4><h4 style="text-align: left;"><span style="font-family: Noto Sans;">O365 Subscription:</span></h4><p><span style="font-family: Noto Sans;">Verify and ensure, the Azure tenant which you have connected has the O365 subscription to access users Calendars / required resource.</span></p><h4 style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></h4><h4 style="text-align: left;"><span style="font-family: Noto Sans;">Reassigning permissions</span></h4><p><span style="font-family: Noto Sans;">Remove and Revoke the permissions and grant it again - simply removing the permission not sufficient, have to revoke the Admin consent and add permission and grant consent again.</span></p><h4 style="text-align: left;"><span style="font-family: Noto Sans;"><br /></span></h4><h4 style="text-align: left;"><span style="font-family: Noto Sans;">Creating a new application</span></h4><p><span style="font-family: Noto Sans;">Delete the existing application and register a new application - if none of the above workarounds resolved the issue, try deleting and existing application and register a new one with correct permission.</span></p><p><span style="font-family: Noto Sans;"><br /></span></p><p><span style="font-family: Noto Sans;">Hope this guide helps to resolve this issue, if you are able to fix this issue using any other method, please mention it in comment section for the benefit of other readers. Thanks!</span></p><p><span style="font-family: Noto Sans;"><br /></span></p><p><span style="font-family: Noto Sans;"><br /></span></p><pre class="has-inner-focus" style="-webkit-font-smoothing: auto; border: 0.125rem solid var(--theme-border); box-sizing: inherit; color: #e6e6e6; font-size: 0.875rem; hyphens: none; line-height: 1.3571; margin-bottom: 0px; margin-top: 0px; outline: none; overflow-wrap: normal; overflow: auto; padding: 1rem; tab-size: 4; word-break: normal;" tabindex="0"><div><span style="box-sizing: inherit; outline-color: inherit;"><span class="hljs-attribute" style="box-sizing: inherit; color: #01cfff; outline-color: inherit;"><span style="font-family: Noto Sans;"><br /></span></span></span></div></pre><p><span style="font-family: Noto Sans;"><br /></span></p><p><span style="font-family: Noto Sans;"><br /></span></p><div><code style="border-radius: 0px; border: 0px; box-sizing: inherit; color: var(--black-800); font-family: var(--ff-mono); font-size: var(--_pr-code-fs); font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline; white-space: inherit;"><br /></code></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com0tag:blogger.com,1999:blog-2027709621250502166.post-89953384128929375002023-01-18T00:06:00.019+05:302023-01-18T00:13:54.424+05:30Streaming Analytics in Google Cloud Platform (GCP) - Building Data Pipeline with Apache Beam<p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiW5GvyfI-7cdkC26yHIms3fDuqqXZMhbBHI3-TuILF0DQu4wiGFj1SYSCKupRdTbKUyVPHH_0DRz5jOKmq9soUrQ90PMZIiGaNg6gAHq4A2AoHJQEfz-YhdJnVBCPIIu8Ke843-7CNP34TZtGQw5nph-L7eeypEi6tEKhRJCUAf9Do7U1225bJumDtuA/s2500/pipeline-1566044.png" style="margin-left: auto; margin-right: auto;"><img alt="Building Apache Beam Data Pipeline" border="0" data-original-height="1724" data-original-width="2500" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiW5GvyfI-7cdkC26yHIms3fDuqqXZMhbBHI3-TuILF0DQu4wiGFj1SYSCKupRdTbKUyVPHH_0DRz5jOKmq9soUrQ90PMZIiGaNg6gAHq4A2AoHJQEfz-YhdJnVBCPIIu8Ke843-7CNP34TZtGQw5nph-L7eeypEi6tEKhRJCUAf9Do7U1225bJumDtuA/s16000/pipeline-1566044.png" title="Building Apache Beam Data Pipeline" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Building Apache Beam Data Pipeline (Source: Pixabay) </span></td></tr></tbody></table><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><p><span style="font-family: Ubuntu; font-size: medium;">In introduction article of this series <a href="https://www.rathishkumar.com/2023/01/streaming-analytics-in-google-cloud.html" rel="nofollow" target="_blank">Streaming Analytics in Google Cloud Platform (GCP) - Introduction</a>, we have seen the basics of streaming analytics, its importance and example uses cases, and short introduction about the Google Cloud Services, we will be using to build Streaming Analytics system in Google Cloud Platform.</span></p><p><span></span></p><a name='more'></a><span style="font-family: Ubuntu; font-size: medium;"><br /></span><p></p><p><span style="font-family: Ubuntu; font-size: medium;"><span><span>The second article </span></span><span><a href="https://www.rathishkumar.com/2023/01/streaming-analytics-in-gcp-setting-up-environment.html" rel="nofollow" target="_blank">Streaming Analytics in Google Cloud Platform (GCP) - Setting Up The Environment</a>,</span><span> </span><span><span>covers the instructions for setting up development and deployment environment in Google Cloud Platform. In this current article, we will be building a data pipeline using Apache Beam Python SDK, we will cover the theoretical aspects of Apache Beam pipeline components first and then discuss the relevant code. </span></span></span></p><p><span style="font-family: Ubuntu; font-size: medium;"><span><span><br /></span></span></span></p><h1 style="text-align: left;">What is a Pipeline in data processing?</h1><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">In simple terms, a data pipeline is a series of connected steps or stages through which raw data is transformed or processed in order to extract insights or perform specific tasks. Data pipelines can be used to perform a wide range of tasks such as data ingestion, cleaning and transformation, analysis and visualisation. The data can be either bounded (batch) or unbounded (stream) and typically involves extracting data from one or more sources, cleaning and processing, and then storing the results in a target location.</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><h1 style="text-align: left;">Pipeline in Apache Beam</h1><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">In Apache Beam, a pipeline is created by building a <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph" rel="nofollow" target="_blank">Directed Acyclic Graph (DAG)</a> of data transformation, where the input is passed through a series of steps or transforms, before being output to the desired location.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph" rel="nofollow" target="_blank">Directed Acyclic Graph (DAG)</a> is a graph which consists of nodes (also known as vertices) and edges, where the edges have directions and do not form any cycles or loops and the edges always point in the same direction.<span class="Apple-converted-space"> </span></span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">In a data processing pipeline, nodes represent the different transformations or operations that are applied to the input data and the edges represent the flow of data from one transformation to another.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">As we have seen in the first part - <a href="https://www.rathishkumar.com/2023/01/streaming-analytics-in-google-cloud.html" rel="nofollow" target="_blank">Streaming Analytics in Google Cloud Platform (GCP) - Introduction</a>, Beam provides many SDKs to build a pipeline and it supports various runners to execute the pipeline. The following pipeline reads data from a text file, does some transformation and writes it back to a text file.</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span style="white-space: normal;"># Import the Apache Beam<br /></span><span style="white-space: normal;">import apache_beam as beam</span><span style="white-space: normal;"><br /><br /></span># Create a Pipeline object<span style="white-space: normal;"><br />pipeline = beam.Pipeline()</span><br style="white-space: normal;" /><br style="white-space: normal;" /><span style="white-space: normal;"># Input: Read raw input data from source</span><br style="white-space: normal;" /><span style="white-space: normal;">input_data = pipeline | "read input" >> beam.io.ReadFromText('./input.txt')</span><br style="white-space: normal;" /><br style="white-space: normal;" /><span style="white-space: normal;"># Transformation: Perform a simple transformation</span><br style="white-space: normal;" /><span style="white-space: normal;">transformed_data = input_data | 'transform data' >> beam.Map(lambda x: int(x) * 2)</span><br style="white-space: normal;" /><br style="white-space: normal;" /><span style="white-space: normal;"># Output: write the transformed data to an output file<br /></span><span style="white-space: normal;">transformed_data | 'write output' >> beam.io.WriteToText('output.txt')</span><br style="white-space: normal;" /><br style="white-space: normal;" /><span style="white-space: normal;"># Run the pipeline</span><br style="white-space: normal;" /><span style="white-space: normal;">pipeline.run()</span></span></pre><p class="p1" style="font-family: "Inconsolata for Powerline"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span class="s4" style="color: #89adf2;"><br /></span></p><p class="p1" style="font-family: "Inconsolata for Powerline"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span class="s4" style="color: #89adf2;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">This pipeline reads data from the input.txt file and performs a transformation of converting to integer and multiplying by 2 and finally stores the result in another text file output.txt</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Create text file input.txt and insert a number and run the pipeline by executing this code as follows:</span><br style="font-family: "Inconsolata for Powerline"; font-size: 11px;" /></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><div style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">touch input.txt<br />echo 2 >> input.txt<br />Python first_pipeline.py</span></div></pre><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: left;"><br /></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">This will create an output text file output.txt, lets's read the output file - it should contain 4, lets verify:</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span style="white-space: normal;">cat output.txt</span></span></pre><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">Now, you have successfully created your first pipeline in Apache Beam!!!</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span class="s4" style="color: #89adf2; font-family: Ubuntu; font-size: medium;"></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">Let's go back to our code and review it again, you may have noticed, we have used some variables to store intermediate results. For example, input_data holds the data from the ReadFromtext() class and transformed_data holds Map() class. In Apache Beam, these variables are called pcollections and operations are called ptransforms. Having a good understanding of pcollection and transforms are very important for building a streaming pipeline, let's cover those topics, before moving to the next items.</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: "Helvetica Neue"; font-size: 11px;"><br /></span></p><h1 style="text-align: left;">PCollection in Apache Beam</h1><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">PCollection (Parallel Collection) is an abstract representation of a distributed dataset, that can be processed in parallel. In Apache beam, it is the main dataset for representing a set of data that can be processed in parallel.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">Reading from files, and reading from PubSub subscriptions are examples of creating a pcollection, also, transforming an existing pcollection also will create a new pcollection. Transformations such as filtering, mapping, grouping and aggregation are applied to the pcollection.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">Example pcollection:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><div style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Reading from a file:</span><span style="font-family: Ubuntu; font-size: medium;"><br /></span></div><div style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></div><p style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><span style="white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">input_data = pipeline | "read input" >> beam.io.ReadFromText('./input.txt')</span></span></p><div><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><div style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Pcollection created by applying transformation on an existing pcollection:</span></div><div style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></div><div><p style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><span style="white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">transformed_data = input_data | 'transform data' >> beam.Map(lambda x: int(x) * 2)</span></span></p></div><div><br /></div><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">Some of the properties of pcollection are given below for your reference:</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><b></b></span></p><blockquote><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><b>Immutable</b> - once created, cannot be modified, to modify, apply a transformation on the original pcollection</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">pcollection may be <b>distributed across multiple machines</b> in a distributed processing environment - allowing parallel processing.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">pcollection is not processed until is executed, such as writing to an output file - <b>lazily evaluated</b>.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">pcollection can be <b>bounded</b> (batch - ex: file) or <b>unbounded</b> (stream - ex:pubsub )</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">pcollection can be partitioned into logical chunks called <b>Windows </b>based on event timestamp.</span></p></blockquote><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"></span></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><h1 style="text-align: left;">PTransforms in Apache Beam</h1><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">PTransform (Parallel Transform) in Apache Beam holds the processing logic, it takes a pcollection as input, applies the transformation logic and gives a transformed pcollection as output.</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><span style="white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">pcollection1 -> ptransform1 -> pcollection2 -> ptransform2 -> pcollection3 -> ptransform3</span></span></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">In a data pipeline, input raw data as pcollection goes through multiple ptransform and the final ptransform produce a final pcollection which will be the output for the sink.</span></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">Example:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"># Transformation: Perform a simple transformation</span></p><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">transformed_data <span class="s1">=</span> input_data <span class="s2">|</span> <span class="s1">'</span><span class="s3">transform data</span><span class="s1">'</span> <span class="s2">>></span> beam<span class="s1">.</span><span class="s6">Map</span><span class="s5">(</span><span class="s2">lambda</span><span class="s6"> </span><span class="s7">x</span><span class="s5">:</span><span class="s6"> </span><span class="s8">int</span><span class="s5">(</span><span class="s6">x</span><span class="s5">)</span><span class="s6"> </span><span class="s1">*</span><span class="s6"> </span><span class="s9">2</span><span class="s5">)</span></span></p></pre><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p4" style="font-family: "Inconsolata for Powerline"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">In this example, input pcollection input_data will be processed using Map transform and output will be stored in a pcollection called transformed_data.</span></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">Note in Apache Beam reading from source and writing to sink operations are ptransforms. See the below example:</span></p><p class="p3" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><p class="p4" style="font-family: "Inconsolata for Powerline"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"># Input: Read raw input data from source</p><p class="p6" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">input_data <span class="s1">=</span> pipeline <span class="s2">|</span> <span class="s1">"</span><span class="s3">read input</span><span class="s1">"</span> <span class="s2">>></span> beam<span class="s1">.</span>io<span class="s1">.</span><span class="s4">ReadFromText</span><span class="s5">(</span><span class="s1">'</span><span class="s3">./input.txt</span><span class="s1">'</span><span class="s5">)</span></span></p><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">transformed_data <span class="s2">|</span> <span class="s1">'</span><span class="s3">write output</span><span class="s1">'</span> <span class="s2">>></span> beam<span class="s1">.</span>io<span class="s1">.</span><span class="s6">WriteToText</span><span class="s5">(</span><span class="s1">‘</span><span class="s3">output.txt</span><span class="s1">'</span><span class="s5">)</span></span></p><p class="p11" style="color: #89adf2; font-family: "Inconsolata for Powerline"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br style="white-space: normal;" /></p></pre><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p4" style="font-family: "Inconsolata for Powerline"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"></span></p><blockquote><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">Apache Beam provides many pre-built ptransform with some of the commonly used ptransforms including:</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><b>Map</b>: Applies a function to each element of a PCollection and returns a new PCollection containing the transformed elements.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><b>Filter</b>: Keeps only elements from a PCollection that satisfy a certain predicate and returns a new PCollection containing the filtered elements.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><b>FlatMap</b>: Applies a function to each element of a PCollection and returns a new PCollection containing the transformed elements, which may be of a different size than the input PCollection.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><b>Combine</b>: Aggregates elements of a PCollection using a specific combiner function and returns a single output element.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><b>GroupByKey</b>: Groups the elements of a PCollection by key and returns a PCollection of key-value pairs, where the values are grouped into an iterable object.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"></p></blockquote><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">We have a good understanding of the building blocks of the Apache Beam data pipeline, let's put together all our learning till this point, and design a streaming pipeline in Apache Beam.</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: "Helvetica Neue"; font-size: 11px;"><br /></span></p><h1 style="text-align: left;">Streaming Pipeline in Apache Beam</h1><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">A streaming pipeline is a pipeline that processes data in real-time as it is received at the system. Typically used to process data from streaming sources, such as sensors, logs, social media feeds, financial ticker logs, etc.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">In the streaming pipeline, the input data is typically from an unbounded data source which has an unlimited number of elements and a series of PTransform are applied to an unbounded PCollection.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">Let's build a streaming pipeline in Apache Beam, using the pcollection, and ptransform as discussed above:</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">import<span class="s1"> json</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s2">import</span> apache_beam <span class="s2">as</span> beam</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s2">from</span> apache_beam<span class="s3">.</span>options<span class="s3">.</span>pipeline_options <span class="s2">import</span> PipelineOptions</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s2">from</span> apache_beam<span class="s3">.</span>options<span class="s3">.</span>pipeline_options <span class="s2">import</span> StandardOptions</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s2">from</span> apache_beam<span class="s3">.</span>io<span class="s3">.</span>gcp<span class="s3">.</span>bigquery <span class="s2">import</span> WriteToBigQuery</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><br /># define the Pub/Sub subscription id</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s4">SUBSCRIPTION</span><span class="s1"> </span><span class="s3">=</span><span class="s1"> </span><span class="s3">"</span>projects/projectname/subscriptions/subscriptionname<span class="s3">"</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><br />#Set the BigQuery schema.</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s4">SCHEMA</span><span class="s1"> </span><span class="s3">=</span><span class="s1"> </span><span class="s3">'</span>name:STRING,age:INTEGER<span class="s3">'</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><br /># Set the table name.</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s4">TABLE_NAME</span><span class="s1"> </span><span class="s3">=</span><span class="s1"> </span><span class="s3">'</span>project:dataset.table<span class="s3">'</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s5"><br />def</span><span class="s1"> </span>parse_json_message<span class="s6">(</span><span class="s4">message</span><span class="s6">):</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s1"> </span><span class="s3">"""</span>Parse the input json message <span class="s3">"""</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="Apple-converted-space"> </span>row <span class="s3">=</span> json<span class="s3">.</span><span class="s7">loads</span><span class="s6">(</span><span class="s7">message</span><span class="s6">)</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s1"> </span>return<span class="s1"> </span><span class="s3">{</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="Apple-converted-space"> </span><span class="s3">"</span><span class="s8">name</span><span class="s3">":</span> row<span class="s6">[</span><span class="s3">"</span><span class="s8">name</span><span class="s3">"</span><span class="s6">]</span><span class="s3">,</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="Apple-converted-space"> </span><span class="s3">"</span><span class="s8">age</span><span class="s3">":</span> row<span class="s6">[</span><span class="s3">"</span><span class="s8">age</span><span class="s3">"</span><span class="s6">]</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="Apple-converted-space"> </span><span class="s3">}</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><br /># Define pipeline options - to enable streaming mode</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s1">options </span><span class="s3">=</span><span class="s1"> </span>PipelineOptions<span class="s6">()</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s1">options</span><span class="s3">.</span>view_as<span class="s6">(</span>StandardOptions<span class="s6">)</span><span class="s3">.</span><span class="s1">streaming </span><span class="s3">=</span><span class="s1"> </span><span class="s9">True</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><br /># Create a pipeline</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">pipeline <span class="s3">=</span> beam<span class="s3">.</span><span class="s7">Pipeline</span><span class="s6">()</span></span></p><p class="p5" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s10">pipeline </span><span class="s3">=</span><span class="s10"> beam</span><span class="s3">.</span>Pipeline<span class="s6">(</span><span class="s4">options</span><span class="s3">=</span>options<span class="s6">)</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><br /># Read from a Cloud Pub/Sub topic and create a PCollection</span></p><p class="p6" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">input_pcoll <span class="s3">=</span> pipeline <span class="s5">|</span> <span class="s3">'</span><span class="s10">Read message from PubSub</span><span class="s3">'</span> <span class="s5">>></span> beam<span class="s3">.</span>io<span class="s3">.</span><span class="s7">ReadFromPubSub</span><span class="s6">(</span><span class="s4">subscription</span><span class="s3">=</span><span class="s4">SUBSCRIPTION</span><span class="s6">)</span></span></p><p class="p6" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">message_string <span class="s3">=</span> input_pcoll <span class="s5">|</span> <span class="s3">"</span><span class="s10">UTF-8 bytes to string</span><span class="s3">"</span> <span class="s5">>></span> beam<span class="s3">.</span><span class="s7">Map</span><span class="s6">(</span><span class="s5">lambda</span><span class="s7"> </span><span class="s4">msg</span><span class="s6">:</span><span class="s7"> msg</span><span class="s3">.</span><span class="s7">decode</span><span class="s6">(</span><span class="s3">"</span><span class="s10">utf-8</span><span class="s3">"</span><span class="s6">))</span></span></p><p class="p6" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">json_message <span class="s3">=</span> message_string <span class="s5">|</span> <span class="s3">"</span><span class="s10">Parse JSON messages</span><span class="s3">"</span> <span class="s5">>></span> beam<span class="s3">.</span><span class="s7">Map</span><span class="s6">(</span><span class="s7">parse_json_message</span><span class="s6">)</span></span></p><p class="p6" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">output_pcoll <span class="s3">=</span> json_message <span class="s5">|</span> <span class="s3">'</span><span class="s10">Write to BigQuery table</span><span class="s3">'</span> <span class="s5">>></span> <span class="s7">WriteToBigQuery</span><span class="s6">(</span><span class="s4">table</span><span class="s3">=</span><span class="s4">TABLE_NAME</span><span class="s3">,</span><span class="s7"> </span><span class="s4">schema</span><span class="s3">=</span><span class="s4">SCHEMA</span><span class="s6">)</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><br />#Run the pipeline</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span class="s1">pipeline</span><span class="s3">.</span>run<span class="s6">()</span><span class="s3">.</span>wait_until_finish<span class="s6">()</span></span></p></pre><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p3" style="font-family: "Inconsolata for Powerline"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">The above code reads messages from Pub/Sub subscription (unbounded data source) and writes to BigQuery table (sink). The code defines pcollections, ptransforms and additionally pipeline options, BigQuery streaming options.<span class="Apple-converted-space"> </span></span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">Pipeline options used to configure different aspects of pipeline such as runner and runner specific configurations, GCP project, region, job name, etc. In our code, we have specified <span class="s1" style="color: #99a0cc;">options</span><span class="s3" style="color: #79d5ff;">.</span>view_as<span class="s6" style="color: #89adf2;">(</span>StandardOptions<span class="s6" style="color: #89adf2;">)</span><span class="s3" style="color: #79d5ff;">.</span><span class="s1" style="color: #99a0cc;">streaming </span><span class="s3" style="color: #79d5ff;">=</span><span class="s1" style="color: #99a0cc;"> </span><span class="s9" style="color: #fd8b52;">True<span class="Apple-converted-space"> </span></span>to enable streaming.<span class="Apple-converted-space"> </span></span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">We use BigQueryIO to write messages to BigQuery table, <span class="s7" style="color: #688df5;">WriteToBigQuery </span>transform requires table name, schema, table’s create disposition and tables write disposition.<span class="Apple-converted-space"> </span></span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><b></b></span></p><blockquote style="text-align: left;"><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><b>Table name</b> - name of the destination table </span></p></blockquote><blockquote style="text-align: left;"><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><b>Schema</b> - destination table schema </span></p></blockquote><blockquote style="text-align: left;"><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><b>Create disposition </b>- specifies whether the destination table must exist or can be created by the write operation </span></p></blockquote><blockquote style="text-align: left;"><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><b>Write disposition</b> - specifies whether write will be truncate and write or append rows to an existing table or write only to an empty table</span></p></blockquote><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: left;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><p class="p6" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">output_pcoll<span class="Apple-converted-space"> </span><span class="s3">=</span> json_message <span class="s5">|</span> <span class="s3">'</span><span class="s10">Write to BigQuery table</span><span class="s3">'</span> <span class="s5">>></span> <span class="s7">WriteToBigQuery</span><span class="s6">(</span></span><span class="Apple-converted-space" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;"> <span> </span></span><span class="s4" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">table</span><span class="s3" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">=</span><span class="s4" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">TABLE_NAME</span><span class="Apple-converted-space" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;"> <span> </span><span> </span></span><span class="s4" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">schema</span><span class="s3" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">=</span><span class="s4" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">SCHEMA</span><span class="s3" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">,</span><span class="Apple-converted-space" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;"> </span></p><p class="p6" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; white-space: normal;"><span class="s4" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">write_disposition</span><span class="s3" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">=</span><span style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">beam</span><span class="s3" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">.</span><span style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">io</span><span class="s3" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">.</span><span style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">BigQueryDisposition</span><span class="s3" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">.</span><span class="s4" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">WRITE_TRUNCATE</span><span class="s3" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">,</span><span class="Apple-converted-space" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;"> </span><span class="s4" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">create_disposition</span><span class="s3" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">=</span><span style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">beam</span><span class="s3" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">.</span><span style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">io</span><span class="s3" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">.</span><span style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">BigQueryDisposition</span><span class="s3" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">.</span><span class="s4" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">CREATE_IF_NEEDED</span><span class="s6" style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large;">)</span></p></pre><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p6" style="color: #99a0cc; font-family: "Inconsolata for Powerline"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">Here, table name is a fully-qualified BigQuery table name and it consist of three parts:</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Source Code Pro; font-size: medium;"></span></p><blockquote><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Source Code Pro; font-size: medium;">Project ID: Google Cloud Project ID</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Source Code Pro; font-size: medium;">Dataset ID: BigQuery dataset ID</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Source Code Pro; font-size: medium;">Table ID: BigQuery table ID</span></p><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"></p></blockquote><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">And it is specified as follows:</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><span style="white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">[project_id]:[dataset_id].[table_id]</span></span></p><div><br /></div><h1 style="text-align: left;">Deploying Apache Beam Pipeline on GCP Cloud Dataflow</h1><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">You have successfully build the data pipeline using Apache Beam Python SDK, it is time to deploy the Pipeline on GCP Cloud Dataflow. In the <a href="https://www.rathishkumar.com/2023/01/streaming-analytics-in-google-cloud.html" rel="nofollow" target="_blank">first introductory article</a> we have discussed about Cloud Dataflow, if you have not read that article, it is time to review quickly. To deploy a pipeline as dataflow job, we will be using command line arguments and we will pass following arguments to our pipeline as pipeline options.</span></p><div><br /></div><div><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"><span style="background-color: white; color: #333333;"></span></span></p><blockquote><span style="font-family: Ubuntu; font-size: medium;"><span style="background-color: white; color: #333333;"><b>Jobname</b> - Name of dataflow job<br /></span><span style="color: #333333;"><span style="background-color: white;"><b>Runner</b> - Name of runner (dataflow runner)<br /></span></span><b>Project</b> - GCP project ID<br /><b>Region</b> - GCP Compute Engine region<br /><b>Staging Location</b> - Cloud Storage bucket name for staging binary and temporary files<br /><b>Temporary Location</b> - <span style="background-color: white; color: #333333;">Path for temporary files<br /></span><span style="color: #333333;"><span style="background-color: white;"><b>Network</b> - </span></span><span style="background-color: rgba(255, 255, 255, 0.95); color: #202124;">The Compute Engine </span>network<span style="background-color: rgba(255, 255, 255, 0.95); color: #202124;"> for launching Compute Engine instances </span><span style="background-color: rgba(255, 255, 255, 0.95); color: #202124;">to run your pipeline<br /></span><span style="background-color: rgba(255, 255, 255, 0.95); color: #202124;"><b>Subnetwork</b> - </span><span style="background-color: rgba(255, 255, 255, 0.95); color: #202124;">The Compute Engine </span>subnetwork<span style="background-color: rgba(255, 255, 255, 0.95); color: #202124;"> for launching Compute Engine instances to run your pipeline</span></span></blockquote><span style="font-family: Ubuntu; font-size: medium;"><span style="background-color: rgba(255, 255, 255, 0.95); color: #202124;"></span></span><p></p></div><div><br /></div><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Lets update the Pipeline options in our code:</span></p><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Import the Pipeline option related libraries:</span></p><div><p style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><span style="font-family: Source Code Pro; font-size: medium;"><span style="color: #cfe2f3;"><span style="white-space: normal;">from apache_beam.options.pipeline_options import PipelineOptions<br /></span></span><span style="color: #cfe2f3;"><span style="white-space: normal;">from apache_beam.options.pipeline_options import StandardOptions<br /></span></span><span style="color: #cfe2f3;"><span style="white-space: normal;">from apache_beam.options.pipeline_options import SetupOptions<br /></span></span><span style="color: #cfe2f3;"><span style="white-space: normal;">from apache_beam.options.pipeline_options import GoogleCloudOptions<br /></span></span><span style="color: #cfe2f3;"><span style="white-space: normal;">from apache_beam.options.pipeline_options import WorkerOptions</span></span></span></p></div><br /><br /><br /><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Import Argument Parser to get command line input to our pipeline code:</span></p></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><span style="white-space: normal;">import argparse</span></span></pre></div><div><br /><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Update our pipeline option to get Pub/Sub subscription for input, output_table to specify the BigQuery table name, and other GCP options as listed above.</span></p></div><div style="text-align: left;"><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #cfe2f3; font-family: "Source Code Pro"; font-size: large; white-space: normal;">parser = argparse.ArgumentParser()</span><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><br style="white-space: normal;" /><span style="white-space: normal;">parser.add_argument("--subscription",help="Input PubSub subscription of the form " '"projects/<PROJECT>/subscriptions/<SUBSCRIPTION>."')</span><br style="white-space: normal;" /><span style="white-space: normal;">parser.add_argument("--output_table", help="BigQuery table of the form " '"project:dataset:table"')</span><br style="white-space: normal;" /><span style="white-space: normal;">parser.add_argument('--project',required=True, help='Specify Google Cloud project')</span><br style="white-space: normal;" /><span style="white-space: normal;">parser.add_argument('--region', required=True, help='Specify Google Cloud region')</span><br style="white-space: normal;" /><span style="white-space: normal;">parser.add_argument('--staging_location', required=True, help='Specify Cloud Storage bucket for staging')</span><br style="white-space: normal;" /><span style="white-space: normal;">parser.add_argument('--temp_location', required=True, help='Specify Cloud Storage bucket for temp')</span><br style="white-space: normal;" /><span style="white-space: normal;">parser.add_argument('--runner', required=True, help='Specify Apache Beam Runner')</span><br style="white-space: normal;" /><span style="white-space: normal;">parser.add_argument('--network', required=True, help='Specify network')</span><br style="white-space: normal;" /><span style="white-space: normal;">parser.add_argument('--subnetwork', required=True, help='Specify subnetwork')<br /></span></span><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;"><br style="white-space: normal;" /><span style="white-space: normal;">opts, pipeline_args = parser.parse_known_args(argv)</span><br style="white-space: normal;" /><br style="white-space: normal;" /><span style="white-space: normal;">options = PipelineOptions(pipeline_args)</span><br style="white-space: normal;" /><span style="white-space: normal;">options.view_as(SetupOptions).save_main_session = True</span><br style="white-space: normal;" /><span style="white-space: normal;">options.view_as(StandardOptions).streaming = True</span><br style="white-space: normal;" /><span style="white-space: normal;">options.view_as(GoogleCloudOptions).project = opts.project</span><br style="white-space: normal;" /><span style="white-space: normal;">options.view_as(GoogleCloudOptions).region = opts.region</span><br style="white-space: normal;" /><span style="white-space: normal;">options.view_as(GoogleCloudOptions).staging_location = opts.staging_location</span><br style="white-space: normal;" /><span style="white-space: normal;">options.view_as(GoogleCloudOptions).temp_location = opts.temp_location</span><br style="white-space: normal;" /><span style="white-space: normal;">options.view_as(GoogleCloudOptions).job_name = '{0}{1}'.format('stream-analytics-pipeline',time.time_ns())</span><br style="white-space: normal;" /><span style="white-space: normal;">options.view_as(StandardOptions).runner = opts.runner</span><br style="white-space: normal;" /><span style="white-space: normal;">options.view_as(WorkerOptions).network = opts.network</span><br style="white-space: normal;" /><span style="white-space: normal;">options.view_as(WorkerOptions).subnetwork = opts.subnetwork</span><br style="white-space: normal;" /><span style="white-space: normal;">options.view_as(WorkerOptions).use_public_ips = False</span><br style="white-space: normal;" /><br style="white-space: normal;" /><span style="white-space: normal;">p = beam.Pipeline(options=options)</span></span></pre></div><div><br /></div><div><br /></div><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Change the pipeline transformation code to get subscription name from command line argument:</span></p><div><br /></div><div><p style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><span style="white-space: normal;"><span style="color: #cfe2f3; font-family: Source Code Pro; font-size: medium;">input_pcoll = pipeline | 'Read message from PubSub' >> beam.io.ReadFromPubSub(subscription=opts.subscription)</span></span></p></div><div><br /></div><div><span style="font-family: Ubuntu; font-size: medium;"><br /></span></div><div><span style="font-family: Ubuntu; font-size: medium;">Change the pipeline transformation code to get BigQuery table name from command line argument and add other BigQuery related options as seen above:</span></div><div><span style="font-family: Ubuntu; font-size: medium;"><br /></span></div><div><br /></div><div><p style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><span style="font-family: Source Code Pro; font-size: medium;"><span style="color: #cfe2f3;"><span style="white-space: normal;">output_pcoll = json_message | 'Write to BigQuery table' >> WriteToBigQuery(<br /></span></span><span style="color: #cfe2f3;"><span style="white-space: normal;">table=opts.output_table,<br /></span></span><span style="color: #cfe2f3;"><span style="white-space: normal;">schema=SCHEMA,<br /></span></span><span style="color: #cfe2f3;"><span style="white-space: normal;">write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE,<br /></span></span><span style="color: #cfe2f3;"><span style="white-space: normal;">create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,<br /></span></span><span style="color: #cfe2f3;"><span style="white-space: normal;">method='STREAMING_INSERTS')<br /></span></span><span style="color: #cfe2f3;"><span style="white-space: normal;">output_pcoll = json_message | 'Write to BigQuery table' >> WriteToBigQuery(table=opts.output_table, schema=SCHEMA)</span></span></span></p></div><div><br /></div><div><span style="font-family: Ubuntu; font-size: medium;">Here is our final code:</span></div><div><br /></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #7dcfff;">import</span> json</div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #7dcfff;">import</span> apache_beam <span style="color: #7dcfff;">as</span> beam</div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #7dcfff;">from</span> apache_beam<span style="color: #89ddff;">.</span>options<span style="color: #89ddff;">.</span>pipeline_options <span style="color: #7dcfff;">import</span> PipelineOptions</div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #7dcfff;">from</span> apache_beam<span style="color: #89ddff;">.</span>options<span style="color: #89ddff;">.</span>pipeline_options <span style="color: #7dcfff;">import</span> StandardOptions</div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #7dcfff;">from</span> apache_beam<span style="color: #89ddff;">.</span>options<span style="color: #89ddff;">.</span>pipeline_options <span style="color: #7dcfff;">import</span> SetupOptions</div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #7dcfff;">from</span> apache_beam<span style="color: #89ddff;">.</span>options<span style="color: #89ddff;">.</span>pipeline_options <span style="color: #7dcfff;">import</span> GoogleCloudOptions</div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #7dcfff;">from</span> apache_beam<span style="color: #89ddff;">.</span>options<span style="color: #89ddff;">.</span>pipeline_options <span style="color: #7dcfff;">import</span> WorkerOptions</div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #7dcfff;">from</span> apache_beam<span style="color: #89ddff;">.</span>io<span style="color: #89ddff;">.</span>gcp<span style="color: #89ddff;">.</span>bigquery <span style="color: #7dcfff;">import</span> WriteToBigQuery</div><br style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;" /><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #7dcfff;">import</span> argparse</div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #7dcfff;">import</span> logging</div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #7dcfff;">import</span> time
<div><span style="color: #444b6a; font-style: italic;"># Set the BigQuery schema.</span></div><div><span style="color: #e0af68;">SCHEMA</span> <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">name:STRING,age:INTEGER</span><span style="color: #89ddff;">'</span></div><br /><div><span style="color: #bb9af7;">def</span> <span style="color: #7aa2f7;">parse_json_message</span><span style="color: #9abdf5;">(</span><span style="color: #e0af68;">message</span><span style="color: #9abdf5;">):</span></div><div> <span style="color: #89ddff;">"""</span><span style="color: #9ece6a;">Parse the input json message </span><span style="color: #89ddff;">"""</span></div><div> row <span style="color: #89ddff;">=</span> json<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">loads</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">message</span><span style="color: #9abdf5;">)</span></div><div> <span style="color: #bb9af7; font-style: italic;">return</span> <span style="color: #89ddff;">{</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">name</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> row<span style="color: #9abdf5;">[</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">name</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">]</span><span style="color: #89ddff;">,</span></div><div> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">age</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span> row<span style="color: #9abdf5;">[</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">age</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">]</span></div><div> <span style="color: #89ddff;">}</span></div>
<div><span style="color: #bb9af7;">def</span> <span style="color: #7aa2f7;">run</span><span style="color: #9abdf5;">(</span><span style="color: #e0af68;">argv</span><span style="color: #9abdf5;">):</span></div><div> parser <span style="color: #89ddff;">=</span> argparse<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">ArgumentParser</span><span style="color: #9abdf5;">()</span></div><br /><div> parser<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">add_argument</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">--subscription</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #e0af68;">help</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Input PubSub subscription of the form </span><span style="color: #89ddff;">"</span><span style="color: #7aa2f7;"> </span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">"projects/<PROJECT>/subscriptions/<SUBSCRIPTION>."</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span></div><div> parser<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">add_argument</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">--output_table</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">help</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">BigQuery table of the form </span><span style="color: #89ddff;">"</span><span style="color: #7aa2f7;"> </span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">"project:dataset:table"</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span></div><div> parser<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">add_argument</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">--project</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span><span style="color: #e0af68;">required</span><span style="color: #89ddff;">=</span><span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">help</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">Specify Google Cloud project</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span></div><div> parser<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">add_argument</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">--region</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">required</span><span style="color: #89ddff;">=</span><span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">help</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">Specify Google Cloud region</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span></div><div> parser<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">add_argument</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">--staging_location</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">required</span><span style="color: #89ddff;">=</span><span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">help</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">Specify Cloud Storage bucket for staging</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span></div><div> parser<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">add_argument</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">--temp_location</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">required</span><span style="color: #89ddff;">=</span><span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">help</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">Specify Cloud Storage bucket for temp</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span></div><div> parser<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">add_argument</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">--runner</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">required</span><span style="color: #89ddff;">=</span><span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">help</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">Specify Apache Beam Runner</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span></div><div> parser<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">add_argument</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">--network</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">required</span><span style="color: #89ddff;">=</span><span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">help</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">Specify network</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span></div><div> parser<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">add_argument</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">--subnetwork</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">required</span><span style="color: #89ddff;">=</span><span style="color: #ff9e64;">True</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">help</span><span style="color: #89ddff;">=</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">Specify subnetwork</span><span style="color: #89ddff;">'</span><span style="color: #9abdf5;">)</span></div><br /><br /><div> opts<span style="color: #89ddff;">,</span> pipeline_args <span style="color: #89ddff;">=</span> parser<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">parse_known_args</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">argv</span><span style="color: #9abdf5;">)</span></div><br /><div> options <span style="color: #89ddff;">=</span> <span style="color: #7aa2f7;">PipelineOptions</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">pipeline_args</span><span style="color: #9abdf5;">)</span></div><div> options<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">view_as</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">SetupOptions</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">.</span>save_main_session <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">True</span></div><div> options<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">view_as</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">StandardOptions</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">.</span>streaming <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">True</span></div><div> options<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">view_as</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">GoogleCloudOptions</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">.</span>project <span style="color: #89ddff;">=</span> opts<span style="color: #89ddff;">.</span>project</div><div> options<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">view_as</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">GoogleCloudOptions</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">.</span>region <span style="color: #89ddff;">=</span> opts<span style="color: #89ddff;">.</span>region</div><div> options<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">view_as</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">GoogleCloudOptions</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">.</span>staging_location <span style="color: #89ddff;">=</span> opts<span style="color: #89ddff;">.</span>staging_location</div><div> options<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">view_as</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">GoogleCloudOptions</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">.</span>temp_location <span style="color: #89ddff;">=</span> opts<span style="color: #89ddff;">.</span>temp_location</div><div> options<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">view_as</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">GoogleCloudOptions</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">.</span>job_name <span style="color: #89ddff;">=</span> <span style="color: #89ddff;">'</span><span style="color: #bb9af7;">{0}{1}</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">format</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">'</span><span style="color: #9ece6a;">stream-analytics-pipeline</span><span style="color: #89ddff;">'</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;">time</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">time_ns</span><span style="color: #9abdf5;">())</span></div><div> options<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">view_as</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">StandardOptions</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">.</span>runner <span style="color: #89ddff;">=</span> opts<span style="color: #89ddff;">.</span>runner</div><div> options<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">view_as</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">WorkerOptions</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">.</span>network <span style="color: #89ddff;">=</span> opts<span style="color: #89ddff;">.</span>network</div><div> options<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">view_as</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">WorkerOptions</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">.</span>subnetwork <span style="color: #89ddff;">=</span> opts<span style="color: #89ddff;">.</span>subnetwork</div><div> options<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">view_as</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">WorkerOptions</span><span style="color: #9abdf5;">)</span><span style="color: #89ddff;">.</span>use_public_ips <span style="color: #89ddff;">=</span> <span style="color: #ff9e64;">False</span></div><br /><br /><div> p <span style="color: #89ddff;">=</span> beam<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">Pipeline</span><span style="color: #9abdf5;">(</span><span style="color: #e0af68;">options</span><span style="color: #89ddff;">=</span><span style="color: #7aa2f7;">options</span><span style="color: #9abdf5;">)</span></div><br /><div> <span style="color: #444b6a; font-style: italic;"># Create a pipeline</span></div><div> pipeline <span style="color: #89ddff;">=</span> beam<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">Pipeline</span><span style="color: #9abdf5;">()</span></div><div> pipeline <span style="color: #89ddff;">=</span> beam<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">Pipeline</span><span style="color: #9abdf5;">(</span><span style="color: #e0af68;">options</span><span style="color: #89ddff;">=</span><span style="color: #7aa2f7;">options</span><span style="color: #9abdf5;">)</span></div><br /><div> <span style="color: #444b6a; font-style: italic;"># Read from a Cloud Pub/Sub topic and create a PCollection</span></div><div> input_pcoll <span style="color: #89ddff;">=</span> pipeline <span style="color: #bb9af7;">|</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">Read message from PubSub</span><span style="color: #89ddff;">'</span> <span style="color: #bb9af7;">>></span> beam<span style="color: #89ddff;">.</span>io<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">ReadFromPubSub</span><span style="color: #9abdf5;">(</span><span style="color: #e0af68;">subscription</span><span style="color: #89ddff;">=</span><span style="color: #7aa2f7;">opts</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">subscription</span><span style="color: #9abdf5;">)</span></div><div> string_message <span style="color: #89ddff;">=</span> input_pcoll <span style="color: #bb9af7;">|</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">UTF-8 bytes to string</span><span style="color: #89ddff;">"</span> <span style="color: #bb9af7;">>></span> beam<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">Map</span><span style="color: #9abdf5;">(</span><span style="color: #bb9af7;">lambda</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">msg</span><span style="color: #9abdf5;">:</span><span style="color: #7aa2f7;"> msg</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">decode</span><span style="color: #9abdf5;">(</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">utf-8</span><span style="color: #89ddff;">"</span><span style="color: #9abdf5;">))</span></div><div> json_message <span style="color: #89ddff;">=</span> string_message <span style="color: #bb9af7;">|</span> <span style="color: #89ddff;">"</span><span style="color: #9ece6a;">Parse JSON messages</span><span style="color: #89ddff;">"</span> <span style="color: #bb9af7;">>></span> beam<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">Map</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">parse_json_message</span><span style="color: #9abdf5;">)</span></div><div> output_pcoll <span style="color: #89ddff;">=</span> json_message <span style="color: #bb9af7;">|</span> <span style="color: #89ddff;">'</span><span style="color: #9ece6a;">Write to BigQuery table</span><span style="color: #89ddff;">'</span> <span style="color: #bb9af7;">>></span> <span style="color: #7aa2f7;">WriteToBigQuery</span><span style="color: #9abdf5;">(</span><span style="color: #e0af68;">table</span><span style="color: #89ddff;">=</span><span style="color: #7aa2f7;">opts</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">output_table</span><span style="color: #89ddff;">,</span><span style="color: #7aa2f7;"> </span><span style="color: #e0af68;">schema</span><span style="color: #89ddff;">=</span><span style="color: #e0af68;">SCHEMA</span><span style="color: #9abdf5;">)</span></div><br /><div> <span style="color: #444b6a; font-style: italic;"># Run the pipeline</span></div><div> pipeline<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">run</span><span style="color: #9abdf5;">()</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">wait_until_finish</span><span style="color: #9abdf5;">()
</span><div><span style="color: #bb9af7; font-style: italic;">if</span> <span style="color: #c0caf5;">__name__</span><span style="color: #bb9af7;">==</span><span style="color: #89ddff;">"</span><span style="color: #9ece6a;">__main__</span><span style="color: #89ddff;">"</span><span style="color: #89ddff;">:</span></div><div> logging<span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">getLogger</span><span style="color: #9abdf5;">()</span><span style="color: #89ddff;">.</span><span style="color: #7aa2f7;">setLevel</span><span style="color: #9abdf5;">(</span><span style="color: #7aa2f7;">logging</span><span style="color: #89ddff;">.</span><span style="color: #e0af68;">INFO</span><span style="color: #9abdf5;">)</span></div><div> <span style="color: #7aa2f7;">run</span><span style="color: #9abdf5;">()</span></div></div></div></pre></div><div> </div><div><div><br /></div><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Deploy our code to Google Cloud Dataflow:</span></p><div><br /></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #9d7cd8; font-style: italic;">export</span> BQTABLE=<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">your_big_query_table_name_in_fully_qualified_name</span><span style="color: #89ddff;">"</span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #9d7cd8; font-style: italic;">export</span> SUBSCRIPTION=<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">your_pubsub_subscription</span><span style="color: #89ddff;">"</span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #9d7cd8; font-style: italic;">export</span> PROJECT=<span style="color: #89ddff;">"$(</span><span style="color: #9ece6a;">gcloud config get-value project</span><span style="color: #89ddff;">)"</span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #9d7cd8; font-style: italic;">export</span> BUCKET=<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">gcs_bucket_name</span><span style="color: #89ddff;">"</span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #9d7cd8; font-style: italic;">export</span> REGION=<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">asia-southeast1</span><span style="color: #89ddff;">"</span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #9d7cd8; font-style: italic;">export</span> NETWORK=<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">your_network</span><span style="color: #89ddff;">"</span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #9d7cd8; font-style: italic;">export</span> SUBNETWORK=<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">https://www.googleapis.com/compute/v1/projects/your_project_id/regions/asia-southeast1/subnetworks/your_subnet</span><span style="color: #89ddff;">"</span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #9d7cd8; font-style: italic;">export</span> TEMPLOCATION=<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">gs://</span><span style="color: #c0caf5;">$BUCKET</span><span style="color: #9ece6a;">/tmp</span><span style="color: #89ddff;">"</span></div><div style="color: #a9b1d6; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px;"><span style="color: #9d7cd8; font-style: italic;">export</span> STAGELOCATION=<span style="color: #89ddff;">"</span><span style="color: #9ece6a;">gs://</span><span style="color: #c0caf5;">$BUCKET</span><span style="color: #9ece6a;">/dev</span><span style="color: #89ddff;">"
</span><div>python -m streaming \</div><div> --output_table <span style="color: #89ddff;">"</span><span style="color: #c0caf5;">$BQTABLE</span><span style="color: #89ddff;">"</span> \</div><div> --subscription <span style="color: #89ddff;">"</span><span style="color: #c0caf5;">$SUBSCRIPTION</span><span style="color: #89ddff;">"</span> \</div><div> --runner DataflowRunner \</div><div> --project <span style="color: #89ddff;">"</span><span style="color: #c0caf5;">$PROJECT</span><span style="color: #89ddff;">"</span> \</div><div> --region <span style="color: #89ddff;">"</span><span style="color: #c0caf5;">$REGION</span><span style="color: #89ddff;">"</span> \</div><div> --subnetwork=<span style="color: #89ddff;">"</span><span style="color: #c0caf5;">$SUBNETWORK</span><span style="color: #89ddff;">"</span> \</div><div> --network=<span style="color: #89ddff;">"</span><span style="color: #c0caf5;">$NETWORK</span><span style="color: #89ddff;">"</span> \</div><div> --staging_location=<span style="color: #89ddff;">"</span><span style="color: #c0caf5;">$TEMPLOCATION</span><span style="color: #89ddff;">"</span> \</div><div> --temp_location=<span style="color: #89ddff;">"</span><span style="color: #c0caf5;">$STAGELOCATION</span><span style="color: #89ddff;">"</span> </div></div></pre></div><div><br /></div><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">After few minutes, your can check your job named stream_analytics in Google Cloud Console under Dataflow jobs section.</span></p><div><h1 style="text-align: left;">Summary</h1><p class="p2" style="font-family: "Helvetica Neue"; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">We have covered what is data pipelines in the context of data processing, building data pipeline in Apache Beam and various components of Apache Beam data pipeline such as pcollection, ptransform and pre-built core transforms. Also, we have covered streaming pipeline and defined a streaming pipeline using Apache Beam Python SDK and gone through the pipeline options, PubSub IO and BigQuery IO features. And finally, we have deployed our data pipeline on Google Cloud Dataflow runner. </span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">That’s all for this part, we will introduce new concept in next article of this series. Until then, have a wonderful day!”</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: left;"><br /></p><h2 style="text-align: left;">References</h2><p style="text-align: left;"><span style="font-family: Ubuntu;">Directed Acyclic Graph (DAG): <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">https://en.wikipedia.org/wiki/Directed_acyclic_graph<br /></a>Beam Concepts: <a href="https://beam.apache.org/documentation/">https://beam.apache.org/documentation/<br /></a>Stream Analytics in GCP - introduction: <a href="https://www.rathishkumar.com/2023/01/streaming-analytics-in-google-cloud.html">https://www.rathishkumar.com/2023/01/streaming-analytics-in-google-cloud.html<br /></a>Stream Analytics in GCP - setting up the environment: <a href="https://www.rathishkumar.com/2023/01/streaming-analytics-in-gcp-setting-up-environment.html">https://www.rathishkumar.com/2023/01/streaming-analytics-in-gcp-setting-up-environment.html</a></span></p></div><div><br /></div></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.comtag:blogger.com,1999:blog-2027709621250502166.post-56710159775849723612023-01-13T00:37:00.007+05:302023-01-18T00:10:48.469+05:30Streaming Analytics in Google Cloud Platform (GCP) - Setting Up The Environment<p></p><div style="text-align: justify;"><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1z5wm5JyQtp-hq6hnUtZ6A43LHbea8ktMoylh7yMI2DrZh4IpXctWw-L3IRTpJV9n-se4xFRWgjTrJdc7KQHsQCNiLK20-V652JFcQtzAPvP6-__NSTrPkYWHKU_NwZ0o50dDcBS1Mny8ceHhStrX70gmizkJm3ARPyX4LqOpQsbyrSRwlhm6ibw1Ww/s4000/cloud-computing-2001090.jpg" style="margin-left: auto; margin-right: auto;"><img alt="Streaming Analytics in GCP" border="0" data-original-height="2250" data-original-width="4000" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1z5wm5JyQtp-hq6hnUtZ6A43LHbea8ktMoylh7yMI2DrZh4IpXctWw-L3IRTpJV9n-se4xFRWgjTrJdc7KQHsQCNiLK20-V652JFcQtzAPvP6-__NSTrPkYWHKU_NwZ0o50dDcBS1Mny8ceHhStrX70gmizkJm3ARPyX4LqOpQsbyrSRwlhm6ibw1Ww/s16000/cloud-computing-2001090.jpg" title="Streaming Analytics in GCP" /></a></td></tr><tr><td class="tr-caption"><div style="text-align: center;"><span style="font-size: small;">Streaming Analytics in GCP (Source: Pixabay) </span></div><div style="font-size: small; text-align: center;"><br /></div></td></tr></tbody></table></div><p></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-family: Ubuntu; font-size: medium;">Hello everyone, in the previous article <a href="https://www.rathishkumar.com/2023/01/streaming-analytics-in-google-cloud.html" rel="nofollow" target="_blank">Streaming Analytics in Google Cloud Platform - Introduction</a>, we have covered what is streaming analytics, what services we are going to use and a quick introduction to each service. In this part of the series, we will begin the installation of SDKs, and libraries and set up our environment.</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span></span></p><a name='more'></a><span style="font-family: Ubuntu; font-size: medium;"><br /></span><p></p><p style="height: 0px;"></p><p></p><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: justify;"><tbody><tr><td class="tr-caption"><div style="text-align: left;"><br /></div><div style="text-align: left;"><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">Having a well-configured development environment is super crucial down the road. I have seen many questions in the stack overflow and other online forums asking questions related to </span><span style="font-family: Roboto; font-size: medium;"><i>“module not found”, “unable to import packages”, “access denied”. “resource not found”, etc</i></span><span style="font-family: Ubuntu; font-size: medium;">. The problem is not, getting errors and fixing them by asking online, in fact, it is a good practice to ask someone who knows and do research on our own to find solutions to our technical problems, if you look back, usually, the problems which you spend many numbers of hours researching are the ones still in your mind and maybe forever. In my first job, I spent a huge amount in asking questions and researching related to setting up MySQL replication, and the knowledge gained from those times is still fresh in my mind, even after many years. So asking questions is good and doing your own research is very important. </span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;">The problem is the time! You will be spending a lot of time on these kinds of questions and waiting for answers, which could be avoided if you simply spend a few minutes setting up your development correctly. </span><span style="font-family: Ubuntu; font-size: medium;">I assume you get a clear understanding of why this part is very important, let's begin with installations.</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p></div><h1><span style="font-family: Ubuntu;">Google Cloud Command-line SDK (gcloud)</span></h1><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-family: Ubuntu; font-size: medium;"><a href="https://cloud.google.com/sdk/gcloud" rel="nofollow" target="_blank">gcloud is a command line interface (CLI)</a> tool for the Google Cloud Platform (GCP). It is powerful and flexible, allows you to use the command line to perform various tasks on GCP, such as creating and managing services and interacting with services, and deploying applications. It is a very essential tool for developers working with GCP.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-family: Ubuntu; font-size: medium;">I am going to cover the installation of gcloud CLI for MacOS, official documentation contains the detailed instruction for other operating systems, please refer here if you are using Windows.</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="text-align: center;"><span style="font-family: Ubuntu; font-size: medium;">Create a folder in a preferred location</span></span></p><p class="p1" style="font-family: "Helvetica Neue"; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><div style="text-align: left;"><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;"><span class="nb" style="box-sizing: border-box; color: #658b00;">mkdir</span> gcp
cd gcp</span></code></pre></div><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Download the gcloud SDK</span></p><div style="text-align: left;"><div class="language-console highlighter-rouge" style="background-color: #031f30; box-sizing: border-box; color: #d3d4d4; margin: 15px 0px;"><div class="highlight" style="box-sizing: border-box;"><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-412.0.0-darwin-x86_64.tar.gz</span></code></pre></div></div></div><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Extract the gcloud archive in the current directory</span></p><div style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"><div style="font-family: Times; font-size: medium;"><div class="language-console highlighter-rouge" style="background-color: #031f30; box-sizing: border-box; color: #d3d4d4; margin: 15px 0px;"><div class="highlight" style="box-sizing: border-box;"><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">tar -xzvf google-cloud-cli-412.0.0-darwin-x86_64.tar.gz</span></code></pre></div></div></div></span></div><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Run the installer script to start the gcloud SDK installation</span></p><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><div style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">./google-cloud-sdk/install.sh</span></code></pre></span></div><p style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;">Once the installation completed, you can start a new terminal window, so that the changes take effect. </span><span style="font-family: Ubuntu; font-size: medium; text-align: justify;">To verify the installation, run the following command, it will show the similar output as shown below for successful installation</span></p><p><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p></p><div style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">gcloud version</span></code></pre></span></div><div style="text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></div><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"></span></p><div class="separator" style="clear: both; text-align: center;"><span style="font-family: Ubuntu; font-size: medium;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaFJi13QvzqlEQ_8e9gGyjtGXM-mvuqUMmbIMluwlPjN-Z0-3bpbc3_KvFE1w4TFS84kyigp9osG3hafHOA28CjDykzvvNTbovo9pA7p4xDtQlxKMbeEp6PZ_K-8Y01zt9aSS7cUEv_j-zuewyU1GJiHOQ7Ink3wp5xZLgXOnJExSWqZY7k4Ycysw6UA/s2064/version.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="gcloud command line interface sdk" border="0" data-original-height="1130" data-original-width="2064" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaFJi13QvzqlEQ_8e9gGyjtGXM-mvuqUMmbIMluwlPjN-Z0-3bpbc3_KvFE1w4TFS84kyigp9osG3hafHOA28CjDykzvvNTbovo9pA7p4xDtQlxKMbeEp6PZ_K-8Y01zt9aSS7cUEv_j-zuewyU1GJiHOQ7Ink3wp5xZLgXOnJExSWqZY7k4Ycysw6UA/s16000/version.jpg" title="gcloud CLI SDK" /></a></span></div><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: left;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span style="font-family: Ubuntu; font-size: medium;">To initialise the gcloud, run the following command</span></p><p class="p1" style="font-family: "Helvetica Neue"; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span style="font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">gcloud init</span></code></pre><p class="p1" style="font-family: "Helvetica Neue"; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span style="font-family: Ubuntu; font-size: medium;">Init command will launches as interactive getting started workflow for the gcloud command line. This command will authorise the gcloud and other SDKs to access to Google Cloud resources and set the current project. This step is must to complete authentication and google cloud resources.</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span style="font-family: Ubuntu;"><span style="font-size: medium;">Few useful commands to try:</span><br /></span></p><p class="p1" style="font-family: "Helvetica Neue"; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span style="font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">gcloud init</span></code></pre><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: left; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">gcloud version - Print version information for Google Cloud CLI components
gcloud info - Display information about the current gcloud environment
gcloud help - Search gcloud help text
gcloud cheat-sheet - Display gcloud cheat sheet
</span></code></pre><p class="p1" style="font-family: "Helvetica Neue"; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span style="font-size: medium;"><br /></span></p><p><span style="font-family: Ubuntu;"><span style="font-size: medium;">Example output:</span></span></p><span style="font-family: Ubuntu;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqR9TNwKAOsuwAIebB-7eczQsZ0KiFBBeR-h8I6lLJ8B5WZJ8YxGIKuImtSx84GyIJxEiaR_t63o4Qm79dzTjQPv_IMK64rZdLlRF8C9Lqtj7mFCTChNwyu2FwLhrVoW16juPsZkbbmBgK2zl1CUTAHT43UNl7AUFl2UqL-VuavKKmvLf_cXWXhsQmWA/s3808/gcloud%20cheat-sheet.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="gcloud CLI SDK - cheat-sheet" border="0" data-original-height="2256" data-original-width="3808" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqR9TNwKAOsuwAIebB-7eczQsZ0KiFBBeR-h8I6lLJ8B5WZJ8YxGIKuImtSx84GyIJxEiaR_t63o4Qm79dzTjQPv_IMK64rZdLlRF8C9Lqtj7mFCTChNwyu2FwLhrVoW16juPsZkbbmBgK2zl1CUTAHT43UNl7AUFl2UqL-VuavKKmvLf_cXWXhsQmWA/s16000/gcloud%20cheat-sheet.jpg" title="gcloud CLI SDK - cheat-sheet" /></a></div></span><h1><span style="font-family: Ubuntu;">Apache Beam Python SDK</span></h1><span style="font-family: Ubuntu;"><br /><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><a href="https://beam.apache.org/get-started/quickstart-py/" rel="nofollow" target="_blank">Apache Beam Python SDK</a> support Python 3.6, 3.7 and 3.8. I will be using Python 3.9 - at the time of writing this article, Apache Beam Python SDK does not support Python 3.10. So I am using 3.9 but please note you might not see this 3.9 support information in official documentation and as it is a continuously evolving project, document may be updated any time. Safer options are Python 3.8, for our implementation, Python 3.9 works fine.</span></p></span><p style="text-align: justify;"><span style="text-align: start;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></span></p><p style="text-align: justify;"><span style="text-align: start;"><span style="font-family: Ubuntu; font-size: medium;">To check your current Python version:</span></span></p><p style="text-align: justify;"><span style="text-align: start;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></span></p><span style="font-family: Ubuntu;"><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: justify; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">python --version</span></code></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: left;"><span style="font-size: medium;">And you need PIP installer to install Apache Beam package, if you do not have one, please install it, steps available here: <a href="https://pip.pypa.io/en/stable/installation/"><span class="s2">https://pip.pypa.io/en/stable/installation/</span></a></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: left;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">If you are using anaconda, there is a high chance that, PIP is already installed, you can check it by running below command:<br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: justify; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">pip --version</span></code></pre><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">Create a virtual environment for stream analytics project by running below command:</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: justify; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">python -m venv streamanalytics</span></code></pre><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">And activate your environment by running</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: justify; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">source streamanalytics/bin/activate</span></code></pre><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">If you are using anaconda, you can create virtual environment by running below command:</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: justify; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">conda create -n streamanalytics python=3.9</span></code></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: large;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;">To activate conda environment</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: justify; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">conda activate streamanalytics</span></code></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: large;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;">Quick note on virtual environments in Python:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span style="font-size: medium;"></span></p><blockquote><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">Virtual environment used to isolate specific Python environments on single machine, allowing you to work on multiple projects with different packages and package versions, without conflict.<span class="Apple-converted-space"> </span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">For example, in project A, you may need Fast API version 1.0 and project B requires Fast API version 2.0. If you simply install without virtual environment, packages will be installed in global python environment, where you can have either version 1.0 or version 2.0, you cannot have both, by isolating project A packages and project B packages, you can have both versions.<span class="Apple-converted-space"> </span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">Similarly, you may need different version of Pythons in your machines, overwriting/modifying system files when trying to install different Python version in global environment may leads to broken system, where some other functionality which requires the existing specific version may not work.<span class="Apple-converted-space"> </span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">And when you want to ensure everyone in your team are working with the same package versions, you can use virtual environment to reproduce the environment and share with others.</span></p></blockquote><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: start;"><span style="font-size: medium;"></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">Download and install the Apache Beam packages:</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">pip install apache-beam[gcp]</span></code></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"></span></p><blockquote><span style="font-size: medium;">You may have seen only pip install apache-beam instead of apache-beam[gcp] in many documentation. The reason we need apache-beam[gcp] and the difference is Apache Beam[GCP] is a specific implementation of Apache Beam that allows you to run data pipelines on Google Cloud Platform. It provides in-built integration with Google Cloud service such as Cloud Data Runner, BigQuery, Cloud Storage, etc.</span></blockquote><p></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">There is one more thing, to define our pipeline in programming, we need a text editor, I am using VS code, you can feel free to use any editor you are comfortable with.<span class="Apple-converted-space"> </span>We are done with package installations and setting up the isolated virtual environment to run our pipeline, let us now create Project in Google Cloud Platform, create Cloud Pub/Sub Topic and Subscription using gcloud command-line tool and enable required access next.</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;">Google Cloud Project</span></h3><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects" rel="nofollow" target="_blank">Creating a new project</a> for our streaming analytics project:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">gcloud projects create streaming-analytics</span></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;">List project - this command will list Project ID, Project Name and Project Number, note down the Project ID of streaming-analytics project.</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">gcloud projects list</span></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;">Set streaming-analytics project as our current project:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">gcloud config set project streaming-analytics</span></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;">To verify the above steps and view the current project and user, run the below command:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">gcloud config list</span></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;">You must have below permission or <b>Project Creator</b> role to create project on Google Cloud:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">resourcemanager.projects.create</span></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;">Cloud Pub/Sub - Topic & Subscription</span></h3><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><a href="https://cloud.google.com/pubsub/docs/admin" rel="nofollow" target="_blank">Create a Pub/Sub topic</a>:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">gcloud pubsub topics create analytics-topic</span></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><a href="https://cloud.google.com/pubsub/docs/create-subscription" rel="nofollow" target="_blank">Create a Pub/Sub subscription</a> and assign it to above created topic:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><div><strong style="box-sizing: border-box;"><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-weight: 400; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">gcloud pubsub subscriptions create analytics-subscription --topic analytics-topic</span></pre></strong></div><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"></span></p><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><span style="font-size: medium;">Cloud Pub/Sub - Publishing & Pulling Messages</span></h3><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;">To publish a test message run the following command:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">gcloud pubsub topics publish analytics-topic <span>--message '{"name":"streamanalytics", "age":2}'</span>
</span></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;">This message will be sent to Pub/Sub subscription queue, you can pull the message by running following command:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">gcloud pubsub subscriptions pull analytics-subscription --format="json(ackId, message.attributes, message.data.decode(\"base64\").decode(\"utf-8\"), message.messageId, message.publishTime)"</span></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;">Sample output:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjTE-kqdmPEPUf_2iq7IVIxyKgPJYhevQ6ZEqlIe_P_LUkHwIk2sVpZ6N5zWXzW_CnraTp4hGKZd4lTqEJhvQs76t8R3Wp8izmUbIGbkGRhQf2uD1BuNUFBxY51vUKML0Km9wy99wRlYuXJ0FDQWDDacHQVNP52v8w8IKnt2iB-EvQJ5-fNX4Q7It73bA" style="margin-left: 1em; margin-right: 1em;"><img alt="gcloud pubsub publish and receive messages" data-original-height="520" data-original-width="1946" src="https://blogger.googleusercontent.com/img/a/AVvXsEjTE-kqdmPEPUf_2iq7IVIxyKgPJYhevQ6ZEqlIe_P_LUkHwIk2sVpZ6N5zWXzW_CnraTp4hGKZd4lTqEJhvQs76t8R3Wp8izmUbIGbkGRhQf2uD1BuNUFBxY51vUKML0Km9wy99wRlYuXJ0FDQWDDacHQVNP52v8w8IKnt2iB-EvQJ5-fNX4Q7It73bA=s16000" title="gcloud pubsub publish and receive messages" /></a></div><br /></span><p><span style="font-family: Ubuntu; font-size: medium;">You must have below permissions/roles assigned to create Cloud Pub/Sub topics, subscription and publish and consume messages:</span></p><span style="font-family: Ubuntu;"><br /><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">roles/pubsub.publisher
roles/pubsub.subscriber
roles/pubsub.viewer</span></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;">BigQuery - Creating Dataset</h3><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p></span><p><span style="font-family: Ubuntu; font-size: medium;"><a href="https://cloud.google.com/bigquery/docs/datasets-intro" rel="nofollow" target="_blank">Create dataset in BigQuery</a>: BigQuery dataset can be created by using many client tools, we will be using the Google Cloud Console. For other options, refer the <a href="https://cloud.google.com/bigquery/docs/datasets#create-dataset" rel="nofollow" target="_blank">official documentation</a>.</span></p><p><span style="font-family: Ubuntu; font-size: medium;">Go to <a href="https://bigquery.cloud.google.com/">BigQuery Console</a> -> Click on View Action (next to project name) -> Create Dataset</span></p><p><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p><span style="font-family: Ubuntu; font-size: medium;"></span></p><div class="separator" style="clear: both; text-align: center;"><span style="font-family: Ubuntu; font-size: medium;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiIy3VVN8BqUHKKGB-BjqLeV454bWHJobJIoThf8OLwmQM62v3QnMIioR-fo08jEmNErXwkoOOXIvS4DN2-kWQHITpxn29M-12fzrzruhIdBtaY5ceCZtxJk5srvJJqKJEYS9LWsyDTONgrcbfRyO4BsVNVp0efIYzg0xLv1rLoZg2DStqJfwCP_V46SQ" style="margin-left: 1em; margin-right: 1em;"><img alt="BigQuery Create Dataset" data-original-height="1628" data-original-width="1130" height="640" src="https://blogger.googleusercontent.com/img/a/AVvXsEiIy3VVN8BqUHKKGB-BjqLeV454bWHJobJIoThf8OLwmQM62v3QnMIioR-fo08jEmNErXwkoOOXIvS4DN2-kWQHITpxn29M-12fzrzruhIdBtaY5ceCZtxJk5srvJJqKJEYS9LWsyDTONgrcbfRyO4BsVNVp0efIYzg0xLv1rLoZg2DStqJfwCP_V46SQ=w444-h640" title="BigQuery Create Dataset" width="444" /></a></span></div><span style="font-family: Ubuntu; font-size: medium;"><br /></span><p><span style="font-family: Ubuntu; font-size: medium;">Enter Dataset ID, Data Location, Other options and click on Create Dataset.</span></p><span style="font-family: Ubuntu; font-size: medium;"><br /></span><p></p><br /><h3 style="font-family: Ubuntu; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;">BigQuery - CREATE & SELECT Table</h3><div><br /></div><div><span style="font-family: Ubuntu; font-size: medium;"><br /></span></div><p><span style="font-family: Ubuntu; font-size: medium;">In BigQuery, <a href="https://cloud.google.com/bigquery/docs/tables#create-table" rel="nofollow" target="_blank">table can be created in many ways</a>, let us create with standard SQL as below and run the BigQuery Editor:</span></p><div><span style="font-family: Ubuntu; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">CREATE TABLE streamanalytics.user(
<span> </span>name STRING,
<span> </span>age INTEGER
);</span></pre></div><div><span style="font-family: Ubuntu; font-size: large;"><br /></span></div><p><span style="font-family: Ubuntu; font-size: medium;">In streaming analytics pipeline, we can create tables and populate data from the data pipeline, to setup the environment correctly, we are creating in console and verifying the access.</span></p><p><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p><span style="font-family: Ubuntu; font-size: medium;">Populate table with sample data:</span></p><p><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">INSERT INTO `streamanalytics.user` VALUES ("stream",1);</span></pre><p><span style="font-family: Ubuntu; font-size: medium;">To query user table:</span></p><p><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">SELECT * FROM `streamanalytics.user`;</span></pre></div><div><br /></div><p><span style="font-family: Ubuntu; font-size: medium;">You must have below roles to create dataset & tables, insert records and select records.</span></p><div><span style="font-family: Ubuntu; font-size: large;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">BigQuery Admin OR BigQuery Editor OR BigQuery Data Owner </span></pre></div><div><br /></div><div><h4 class="role-title add-link" data-text="BigQuery Data Owner " id="bigquery.dataOwner" role="presentation" style="box-sizing: inherit; color: #202124; font-family: Roboto, "Noto Sans", "Noto Sans JP", "Noto Sans KR", "Noto Naskh Arabic", "Noto Sans Thai", "Noto Sans Hebrew", "Noto Sans Bengali", sans-serif; font-size: 14px; line-height: inherit; margin-inline-end: -40px; margin: 0px; overflow: hidden; padding-inline-end: 40px; padding-left: 0px; padding-right: 0px; text-align: left; text-overflow: ellipsis;"><br /></h4><div><h3 style="font-family: Ubuntu; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;">Cloud Dataflow Jobs:</h3></div><div><br /></div><p><span style="font-family: Ubuntu; font-size: medium;">To create and deploy Cloud Dataflow jobs, you need the below permission/roles assigned:</span></p><p><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><h4 class="role-title add-link" data-text="BigQuery Data Owner " id="bigquery.dataOwner" role="presentation" style="box-sizing: inherit; color: #202124; font-family: Roboto, "Noto Sans", "Noto Sans JP", "Noto Sans KR", "Noto Naskh Arabic", "Noto Sans Thai", "Noto Sans Hebrew", "Noto Sans Bengali", sans-serif; font-size: 14px; line-height: inherit; margin-inline-end: -40px; margin: 0px; overflow: hidden; padding-inline-end: 40px; padding-left: 0px; padding-right: 0px; text-align: left; text-overflow: ellipsis;"><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: black; font-weight: 400; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; text-align: justify; word-break: normal;"><span style="color: #c1f1f0; font-size: medium;">roles/dataflow.admin</span></pre></h4><h4 class="role-title add-link" data-text="BigQuery Data Owner " id="bigquery.dataOwner" role="presentation" style="box-sizing: inherit; color: #202124; font-family: Roboto, "Noto Sans", "Noto Sans JP", "Noto Sans KR", "Noto Naskh Arabic", "Noto Sans Thai", "Noto Sans Hebrew", "Noto Sans Bengali", sans-serif; font-size: 14px; line-height: inherit; margin-inline-end: -40px; margin: 0px; overflow: hidden; padding-inline-end: 40px; padding-left: 0px; padding-right: 0px; text-align: left; text-overflow: ellipsis;"><br /></h4><div><h3 style="font-family: Ubuntu; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;"><br /></h3><h3 style="font-family: Ubuntu; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px;">Cloud Dataflow Service Account:</h3></div><h4 class="role-title add-link" data-text="BigQuery Data Owner " id="bigquery.dataOwner" role="presentation" style="box-sizing: inherit; color: #202124; line-height: inherit; margin-inline-end: -40px; margin: 0px; overflow: hidden; padding-inline-end: 40px; padding-left: 0px; padding-right: 0px; text-align: left; text-overflow: ellipsis;"><p><span style="font-family: Ubuntu; font-size: medium;"><span style="font-weight: 400;"><br /></span></span></p><p><span style="font-family: Ubuntu; font-size: medium;"><span style="font-weight: 400;">Dataflow uses following two <a href="https://cloud.google.com/dataflow/docs/concepts/security-and-permissions" rel="nofollow" target="_blank">service accounts</a>:</span></span></p><p><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p><span style="font-family: Ubuntu; font-size: medium;">Dataflow Service Account<span style="font-weight: 400;"> - orchestrate dataflow environment, interacts between your project and Dataflow. It is used for worker creation and monitoring. Requires </span>Dataflow Service Agent <span style="font-weight: 400;">role.</span></span></p><p><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p><span style="font-family: Ubuntu; font-size: medium;">Controller Service Account <span style="font-weight: normal;">- used by the workers to access resources needed by the pipeline, if none specified, it will use the compute engine default service account. You need to assign BigQuery and other services roles to this service account to access it from Dataflow workers.</span></span></p><p><span style="font-family: Ubuntu; font-size: medium;"><span style="font-weight: normal;"><br /></span></span></p></h4><p style="box-sizing: inherit; color: #202124; line-height: inherit; margin-inline-end: -40px; margin: 0px; overflow: hidden; padding-inline-end: 40px; padding-left: 0px; padding-right: 0px; text-align: left; text-overflow: ellipsis;"><span style="font-family: Ubuntu; font-size: medium;"><br /></span></p><p style="box-sizing: inherit; color: #202124; line-height: inherit; margin-inline-end: -40px; margin: 0px; overflow: hidden; padding-inline-end: 40px; padding-left: 0px; padding-right: 0px; text-align: left; text-overflow: ellipsis;"><span style="font-family: Ubuntu; font-size: medium;">That’s it for today, we have covered installation and setting up the environment for developing and deploying our pipeline, created Cloud Pub/Sub topic and subscription for publishing and consuming messages, reviewed the permission to create projects, pipelines, BigQuery dataset and tables. We will jump into building our first data pipeline in Apache Beam in next session. Have a good day!</span></p></div><span style="font-family: Ubuntu;"><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><h2 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;">References</h2><div><br /></div><div><br /></div></span><p><span><span style="font-family: Ubuntu; font-size: medium;">Introduction Article: <a href="https://www.rathishkumar.in/2023/01/streaming-analytics-in-google-cloud.html">https://www.rathishkumar.com/2023/01/streaming-analytics-in-google-cloud.html</a></span></span></p><p><span style="font-size: medium;"><span style="font-family: Ubuntu;">gcloud overview: </span><span style="font-family: Ubuntu;"><a href="https://cloud.google.com/sdk/gcloud">https://cloud.google.com/sdk/gcloud</a></span></span></p><p><span style="font-family: Ubuntu; font-size: medium;">gcloud cheatsheet: <a href="https://cloud.google.com/sdk/docs/cheatsheet">https://cloud.google.com/sdk/docs/cheatsheet</a></span></p><p><span style="font-family: Ubuntu; font-size: medium;">Apache Beam Python SDK: <a href="https://beam.apache.org/get-started/quickstart-py/">https://beam.apache.org/get-started/quickstart-py/</a></span></p><p><span style="font-family: Ubuntu; font-size: medium;"><span>GCP Projects: </span><a href="https://beam.apache.org/get-started/quickstart-py/">https://cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy#projects</a></span></p><span style="font-family: Ubuntu; font-size: medium;"><p><span>Cloud Pub/Sub How-to: <a href="https://cloud.google.com/pubsub/docs/how-to">https://cloud.google.com/pubsub/docs/how-to</a></span></p><p><span>BigQuery How-to: </span><a href="https://cloud.google.com/bigquery/docs/how-to">https://cloud.google.com/bigquery/docs/how-to</a></p></span><p><span style="font-family: Ubuntu; font-size: medium;">Cloud Dataflow How-to: <a href="https://cloud.google.com/dataflow/docs/how-to">https://cloud.google.com/dataflow/docs/how-to</a></span></p><span style="font-family: Ubuntu;"><br /></span></td></tr></tbody></table>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.comtag:blogger.com,1999:blog-2027709621250502166.post-87920782390499212712023-01-10T23:18:00.011+05:302023-01-18T00:10:20.490+05:30Streaming Analytics in Google Cloud Platform (GCP) - Introduction<h4 style="text-align: left;"></h4><h3 style="text-align: center;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-ZgRiwbNZEUglPY7_4ocvpy_7ndOlwzAlbM8I6Pwd3V_IGVx_ENwc9r2JXxW8mxIRZnybW1hyTRpuOz6n9QTSoTAd8cJKJdA6pxQUTvTMMJHLXFposlKbvWelEX63VkuOEUJrnOUn9XppfLS6pyKioPyHmZGVs1ETTamZ8J5NIR01z2QSfUKzcC-N7A/s6000/analytics-3088958.jpg" style="margin-left: auto; margin-right: auto;"><span style="font-family: Ubuntu;"><img alt="Streaming Analytics in Google Cloud Platform" border="0" data-original-height="4000" data-original-width="6000" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-ZgRiwbNZEUglPY7_4ocvpy_7ndOlwzAlbM8I6Pwd3V_IGVx_ENwc9r2JXxW8mxIRZnybW1hyTRpuOz6n9QTSoTAd8cJKJdA6pxQUTvTMMJHLXFposlKbvWelEX63VkuOEUJrnOUn9XppfLS6pyKioPyHmZGVs1ETTamZ8J5NIR01z2QSfUKzcC-N7A/s16000/analytics-3088958.jpg" title="Streaming Analytics in Google Cloud Platform" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-family: Ubuntu; font-size: x-small;">Streaming Analytics in Google Cloud Platform (image source - pixabay) </span></td></tr></tbody></table><span style="font-family: Ubuntu;"><br /></span></h3><h2 style="text-align: center;"><span style="font-family: Ubuntu;">From data-to-decision in real-time </span></h2><div><span style="font-family: Ubuntu;"><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">Welcome to our new series on building a streaming analytics system in the Google Cloud Platform!. Let's begin with a quick introduction. Streaming analytics is the process of analysing data in real-time as it is received. Streaming analytics enables an organisation to gain insights and make decisions based on the most up-to-date data, in real time. This is crucial for business as it allows organisations to respond to changes and opportunities in a timely manner.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span></span></p><a name='more'></a><span style="font-size: medium;"><br /></span><p></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">For example, if an organisation is able to detect and respond to changes in customer behaviour or market conditions in real time, it can adjust its strategies and tactics to better meet the needs of its customers and take advantage of emerging opportunities. A streaming analytics system in a retail company can detect when a particular product is becoming more popular among customers and alert the company to this trend in real time. The company is able to respond to this trend by increasing its orders for the product and adjusting its inventory to meet growing customer demand. Also, the company can learn the competitor's actions and adjust product prices dynamically based on the supply. So with real-time insights, a retail company can avoid running out of stock (lost sales) and also takes advantage of the increased demand and dynamic pricing to drive up it is revenue.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span class="Apple-converted-space"><span style="font-size: medium;"> </span></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">I hope this example helps you with understanding the benefits of streaming analytics. There are many other use cases such as fraud detection, performance monitoring, personalisation, etc. where streaming analytics is changing the business. With this basic understanding, in this practical guide, we'll walk you through the process of using Cloud Pub/Sub for data ingestion, Apache Beam pipelines on Dataflow for data processing, BigQuery for storage and analysis, and Looker Studio for data visualisation. Whether you're a seasoned GCP user or new to the platform, this series is designed to provide you with the skills and knowledge you need to build a powerful and efficient streaming analytics system</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">We'll be using the Google Cloud SDK (gcloud CLI tool), Python SDK, and standard SQL to deploy resources, process and analyse data, and more. In the first part of this series, we'll provide a simple introduction to the relevant services, and in the subsequent articles, we'll delve into installing libraries, designing and deploying pipelines, performing aggregations, and more. Let's get started with the introduction!</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><h2 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><b>Cloud Pub/Sub</b></h2><div><b><br /></b></div><div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvO9bfjnblgWXbFNNfycm3eGOMw73k0KBVt5cW1pbpdSBUbFUheQc3SnXxiIzFgmBtG9U3UGsprEG0MxaWDxk-bHkyJQgScqXmjQKf9okb3qjixri9FXTdhF4apIEli2z21kr3G2DngpvtulcXrw9V3MfisCq6JX3tNUByJQk_NQeoX8pYl6nhXdq5XA/s1280/Cloud%20PubSub.jpeg" style="margin-left: auto; margin-right: auto;"><img alt="Google Cloud Pub/Sub" border="0" data-original-height="720" data-original-width="1280" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvO9bfjnblgWXbFNNfycm3eGOMw73k0KBVt5cW1pbpdSBUbFUheQc3SnXxiIzFgmBtG9U3UGsprEG0MxaWDxk-bHkyJQgScqXmjQKf9okb3qjixri9FXTdhF4apIEli2z21kr3G2DngpvtulcXrw9V3MfisCq6JX3tNUByJQk_NQeoX8pYl6nhXdq5XA/w640-h360/Cloud%20PubSub.jpeg" title="Google Cloud Pub/Sub" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Google Cloud Pub/Sub (Source: GCP)</span></td></tr></tbody></table><br /><b><br /></b></div><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><a href="https://cloud.google.com/pubsub/docs/overview" rel="nofollow" target="_blank">Pub/Sub</a> is a messaging service, which allows us to send messages between applications and it is based on the <a href="https://learn.microsoft.com/en-us/azure/architecture/patterns/publisher-subscriber" rel="nofollow" target="_blank">Publisher/Subscriber (pub/sub) pattern</a>. In a distributed system, there will be multiple services running independently, and there must be some way to communicate between these services, and Pub/Sub provides this capacity. There are two components, Topic & Subscription. A message producer application publishes messages on a Topic and one or more subscriber applications express interest in that topic and receive messages through Subscriptions.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">In a Pub/Sub pattern, the publisher and subscriber are independent of each other. Before sending a message, a publisher application does not need to know who is going to/ how many clients are going to receive the messages. And similarly, the subscriber application may not know about the existence of the publisher application. The pub/sub decouples subscribers from the publisher and allows us to develop, deploy and scale the applications independently.<span class="Apple-converted-space"> </span></span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><a href="https://cloud.google.com/pubsub" rel="nofollow" target="_blank">Cloud Pub/Sub</a> is a fully managed service, which means you don’t need to worry about the infrastructure - with just a few clicks, you will be able to create a pub/sub service and ready to send/receive messages. It is used for real-time, many-many, asynchronous messaging.</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><h2 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><b>Apache Beam</b></h2><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigCH6j6Ouukn0qQu-efxFbCKHtrBkCTAf72loQBoC7oJd-RA9Vsf7SVWbRyDnCHrpW10FpKw9CEYh94K-WJ_9OeCDRN65KFbPHdoeTqAWpbI4-bLIGX6rdqX2Xk9xUrJ-5WFMJ1o-WjmlLdy3_pxaY20yTn28ireP9IRuCm8mNwD3zL-d0lr5ttrRLAA/s1000/Beam.png" style="margin-left: auto; margin-right: auto;"><img alt="Apache Beam" border="0" data-original-height="1000" data-original-width="1000" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigCH6j6Ouukn0qQu-efxFbCKHtrBkCTAf72loQBoC7oJd-RA9Vsf7SVWbRyDnCHrpW10FpKw9CEYh94K-WJ_9OeCDRN65KFbPHdoeTqAWpbI4-bLIGX6rdqX2Xk9xUrJ-5WFMJ1o-WjmlLdy3_pxaY20yTn28ireP9IRuCm8mNwD3zL-d0lr5ttrRLAA/w640-h640/Beam.png" title="Apache Beam" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Apache Beam (Source: beam)</span></td></tr></tbody></table><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><br /></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><a href="https://beam.apache.org/" rel="nofollow" target="_blank">Beam</a> stands for <b>B</b>atch + Str<b>eam. </b>It is an <a href="https://beam.apache.org/get-started/beam-overview/" rel="nofollow" target="_blank">open-source unified programming model</a> for defining and executing data pipelines. The main advantage of Beam over other data processing frameworks is its<b> portability</b>.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">First, Apache Beam provides the flexibility to express the data processing logic once and execute it for both batch and stream processing, which means, you write a pipeline once and it is portable to handle both batch and streaming data processing without changing your code.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">Second, you can programmatically define your pipeline using one of the supported languages. Beam provides a number of language-specific SDKs including Java, Python, SQL and Go. This allows you to create your pipeline in any language you are comfortable with, and Beam’s Portability Framework takes care of the execution on runner internally. Also, this enables us to develop cross-language transforms and multi-language pipelines.<span class="Apple-converted-space"> </span></span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">Third, flexibility to run your pipeline in multiple runners (execution environments). The Beam model is designed to be independent of the underlying execution engine, allowing the same Beam pipeline to be executed on different platforms without changing your code. The Beam supports many runners including Google Cloud Dataflow, Apache Flink, Apache Spark and Apache Samza.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">One of the key benefits of the portability provided by Apache Beam is that it can help to avoid vendor lock-in. As the code is not tied to a specific vendor’s technology or platform, it can be migrated to another runtime engine, if necessary. This can provide significant benefits in terms of flexibility, and long-term maintainability of your application. Keep this point in mind, when deciding your next data processing solution.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><h2 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><b>Cloud Dataflow</b></h2><div><b><br /></b></div><div><b><br /></b></div><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixPTFr_SL-SMP6BgunsLYPvq2Vytui5Ra1BuDQhwKPN4VEUVqjgi-jQlMfO2oaPl8zVdMuHjHFMqqKRbetP1ypGRjBjZcAD4Ot4HxaWIesrlvC_FwNSpzM5j--q3B95653yCHOOdlwRz3zNnQHvuXVXfdeEXtJz0kLPk8ziV2r6zk795CUIPJ4w2RRdQ/s438/Data-Flow-Logo.png" style="margin-left: auto; margin-right: auto;"><img alt="GCP Cloud Dataflow" border="0" data-original-height="176" data-original-width="438" height="258" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixPTFr_SL-SMP6BgunsLYPvq2Vytui5Ra1BuDQhwKPN4VEUVqjgi-jQlMfO2oaPl8zVdMuHjHFMqqKRbetP1ypGRjBjZcAD4Ot4HxaWIesrlvC_FwNSpzM5j--q3B95653yCHOOdlwRz3zNnQHvuXVXfdeEXtJz0kLPk8ziV2r6zk795CUIPJ4w2RRdQ/w640-h258/Data-Flow-Logo.png" title="GCP Cloud Dataflow" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">GCP Cloud Dataflow (Source: GCP)</span></td></tr></tbody></table><br /><span style="font-size: medium;"><br /></span><p></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><a href="https://cloud.google.com/dataflow" rel="nofollow" target="_blank">Cloud Dataflow</a> is a fully-managed serverless data processing service. It provides the execution environment for Apache Beam data pipelines.<span class="Apple-converted-space"> </span></span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">Scalability is an important consideration in a distributed system, as a fully-managed service Dataflow takes care of the underlying infrastructure and is designed to handle data processing at scale, it can automatically scale up and down as needed to meet the pipeline demands, and this reduces manual interventions.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">As Dataflow is a part of the Google Cloud Platform, it can be easily integrated with other Google Cloud Services such as Cloud Pub/Sub, BigQuery., etc. This is super helpful when you are building an end-to-end analytics system in the Google Cloud Platform.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">Refer to the <a href="https://cloud.google.com/dataflow#section-10" rel="nofollow" target="_blank">official documentation</a> for complete features provided by Cloud Dataflow.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><h2 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><b>BigQuery</b></h2><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjccTdiaF383ROu-4LI4D2UTJsu6rVgZCojbM3f24l_Ed1112r3k7I9Hp-ry05Vnq1F86CwmjeSvKaH6Ujag_62-N8VV1CyPf_O8GzCX1Gi6XSiQUBi2y2hcY8y-jC49Gzi0pnNSYgaeiaGvQB0HCSP-CmIqn1rO0c6fv1wmRbUXyWPDb3p7OpIxsKEMw/s1442/BigQuery.webp" style="margin-left: auto; margin-right: auto;"><img alt="Google BigQuery" border="0" data-original-height="812" data-original-width="1442" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjccTdiaF383ROu-4LI4D2UTJsu6rVgZCojbM3f24l_Ed1112r3k7I9Hp-ry05Vnq1F86CwmjeSvKaH6Ujag_62-N8VV1CyPf_O8GzCX1Gi6XSiQUBi2y2hcY8y-jC49Gzi0pnNSYgaeiaGvQB0HCSP-CmIqn1rO0c6fv1wmRbUXyWPDb3p7OpIxsKEMw/w640-h360/BigQuery.webp" title="Google BigQuery" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Google BigQuery (Source - GCP)</span></td></tr></tbody></table><br /><span style="font-size: medium;"><br /></span><p></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><a href="https://cloud.google.com/bigquery" rel="nofollow" target="_blank">BigQuery</a> is a serverless data warehouse service provided by the Google Cloud Platform. It is a fully managed service, you simply load your data and start querying it using SQL. You can interact with BigQuery using various tools, a simple one to start with BigQuery web UI, also there are client libraries as well, including bq command line tools and Python, Java and Go client libraries.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">BigQuery is highly scalable and handles large volumes of data efficiently. To achieve low latency and reduce query cost, internally BigQuery uses <a href="https://cloud.google.com/blog/topics/developers-practitioners/bigquery-explained-storage-overview" rel="nofollow" target="_blank">Columnar Storage</a> for storing column-wise data, <a href="https://cloud.google.com/blog/products/bigquery/in-memory-query-execution-in-google-bigquery" rel="nofollow" target="_blank">distributed query execution</a> for shuffling data across multiple workers/nodes and <a href="https://cloud.google.com/bigquery/docs/clustered-tables" rel="nofollow" target="_blank">data partitioning</a>. </span><span style="font-size: medium;"><span>Also, at the user level, you can use partitioning and clustering options to improve performance and reduce costs.</span><span class="Apple-converted-space"> </span></span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">BigQuery provides many interesting features including real-time analytics and built-in machine learning. Also, it is easy to integrate with other Google Cloud services.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">These are the main services, we will be using in this series, we might use a few more services such as Google Cloud Storage, and Cloud Functions, etc. I will add a short introduction to these services at the time of implementation.</span></p><p class="p2" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 12px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;">Part 1 is in the books, but the fun is just getting started. In our next instalment, we'll be tackling the important task of <a href="https://www.rathishkumar.com/2023/01/streaming-analytics-in-gcp-setting-up-environment.html" rel="nofollow" target="_blank">installing the SDKs and client libraries needed for this project</a>. Keep an eye out for it, and have a great day in the meantime!</span></p><p class="p1" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span style="font-size: medium;"><br /></span></p><h2 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><b>References</b></h2><div style="text-align: left;"><br /></div><p style="text-align: left;"><span style="font-size: medium;">Streaming data: <a href="https://en.wikipedia.org/wiki/Streaming_data">https://en.wikipedia.org/wiki/Streaming_data<br /></a></span><span style="font-size: medium;">Publisher-Subscriber pattern: <a href="https://learn.microsoft.com/en-us/azure/architecture/patterns/publisher-subscriber">https://learn.microsoft.com/en-us/azure/architecture/patterns/publisher-subscriber<br /></a></span><span style="font-size: medium;">Cloud Pub/Sub: <a href="https://cloud.google.com/pubsub">https://cloud.google.com/pubsub<br /></a></span><span style="font-size: medium;">Apache Beam: <a href="https://beam.apache.org/about/">https://beam.apache.org/about/<br /></a></span><span style="font-size: medium;">Cloud Dataflow: <a href="https://cloud.google.com/dataflow">https://cloud.google.com/dataflow<br /></a></span><span style="font-size: medium;">BigQuery: <a href="https://cloud.google.com/bigquery">https://cloud.google.com/bigquery<br /></a></span><span style="font-size: medium;">Columnar Storage: <a href="https://cloud.google.com/blog/topics/developers-practitioners/bigquery-explained-storage-overview" rel="nofollow" target="_blank">https://cloud.google.com/blog/topics/developers-practitioners/bigquery-explained-storage-overview</a></span></p></span></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com1tag:blogger.com,1999:blog-2027709621250502166.post-77826509242290027462021-06-03T00:43:00.004+05:302021-06-03T00:44:40.961+05:30Installing and configuring Docker Engine and Docker Compose on CentOS<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><span style="font-size: medium;"><img alt="Installing & Configuring Docker Engine on CentOS" border="0" data-original-height="1221" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSBHmAFjbp2qh3c2LDkmsj6xS3cJf6W-Qjpc34pNwtb1mgI5CEXaPxgN3DorR4BCX30D6dCxSm3kTBZDRRcEOuLx5V7W-CTiQbRoFel2XIiE1Y5Pytc5xDzrd9Zv35a0zeJAMtZsCrcVnI/s16000/ship-6271649_1920.jpeg" style="margin-left: auto; margin-right: auto;" title="Installing & Configuring Docker Engine on CentOS" /></span></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Installing & Configuring Docker Engine & Docker-compose on CentOS. Source: <a href="https://pixabay.com/" rel="nofollow" target="_blank">pixabay</a> </span></td></tr></tbody></table><span style="font-size: medium;"><span><br /><br /><div style="text-align: justify;"><span style="font-family: inherit;">I have written this post as a quick reference guide for installing and configuring Docker Engine and Docker compose on CentOS servers. </span><span style="font-family: inherit;">Knowing the basics of Docker containers helps you focus on the end goal of solving problems, than spending your energy on other less important aspects. </span><span>If you have </span>experimented<span> with multiple packages and applications, you might be knowing that, Docker containers makes it really easy to install and running softwares without worrying about dependencies, scripts and configurations, etc. Also, when we are building and releasing our solutions, it is important to let others consume it in simple process, Docker images helps you achieve that. I will be covering below topics in this article: <span><a name='more'></a></span></span></div></span></span><ul style="text-align: left;"><li><span style="font-family: inherit; font-size: medium;">Supported CentOS Versions</span></li><li><span style="font-family: inherit; font-size: medium;">Setting up Docker Repository</span></li><li><span style="font-family: inherit; font-size: medium;">Installing Docker Engine</span></li><li><span style="font-family: inherit; font-size: medium;">Starting Docker Service</span></li><li><span style="font-family: inherit; font-size: medium;">Verifying Docker Service</span></li><li><span style="font-family: inherit; font-size: medium;">Enabling auto-start for Docker Engine on boot</span></li><li><span style="font-family: inherit; font-size: medium;">Changing the data storage directory (Optional)</span></li><li><span style="font-family: inherit; font-size: medium;">Uninstalling Docker Engine (Optional)</span></li><li><span style="font-family: inherit; font-size: medium;">Installing Docker Compose</span></li><li><span style="font-family: inherit; font-size: medium;">Example docker commands</span></li><li><span style="font-family: inherit; font-size: medium;">Example docker-compose commands</span></li></ul><h2 style="text-align: left;"><span style="font-family: inherit; font-size: large;">Supported CentOS Versions</span></h2><div><span style="font-family: inherit; font-size: medium;">Docker Engine supports CentOS 7 & 8, in this article we will install on CentOS 7, and it will be similar for CentOS 8 as well.</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><h2 style="text-align: left;"><span style="font-family: inherit; font-size: large;">Setting up Docker Repository</span></h2><span style="font-family: inherit; font-size: medium;"><span>When you are installing Docker Engine for the first time in a CentOS machine, need to add Docker Repository. This is a one time activity, if you uninstall Docker Engine for some reason, not required to remove this repo and can be used to install Docker Engine next time from the same repo.</span></span></div><div><br /></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;"><span class="nb" style="box-sizing: border-box; color: #658b00;">sudo </span>yum install <span class="nt" style="box-sizing: border-box; color: #e8a0e8;">-y</span> yum-utils</span></code></pre><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;"><span class="nb" style="box-sizing: border-box; color: #658b00;">sudo </span>yum-config-manager <span class="se" style="box-sizing: border-box; color: #cd5555;">\</span>
<span class="nt" style="box-sizing: border-box; color: #e8a0e8;">--add-repo</span> <span class="se" style="box-sizing: border-box; color: #cd5555;">\</span>
https://download.docker.com/linux/centos/docker-ce.repo</span></code></pre></div><h2 style="text-align: left;"><span style="font-family: inherit; font-size: large;"><br /></span></h2><h2 style="text-align: left;"><span style="font-family: inherit; font-size: large;">Installing Docker Engine</span></h2><div><span style="font-family: inherit; font-size: medium;">Run the below command to install the Docker engine. This will install the latest version of Docker Engine and containerd packages.</span></div><div><div class="language-console highlighter-rouge" style="background-color: #031f30; box-sizing: border-box; color: #d3d4d4; margin: 15px 0px;"><div class="highlight" style="box-sizing: border-box;"><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;"><span class="nb" style="box-sizing: border-box; color: #658b00;">sudo </span>yum install docker-ce docker-ce-cli containerd.io</span></code></pre></div></div></div><h2 style="text-align: left;"><span style="font-family: inherit; font-size: large;"><br /></span></h2><h2 style="text-align: left;"><span style="font-family: inherit; font-size: large;">Starting Docker Service</span></h2><div><span style="font-family: inherit; font-size: medium;">After installing Docker Engine, it will not be started automatically, to start the Docker Engine run the below command</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;"><span class="nb" style="box-sizing: border-box; color: #658b00;">sudo </span>systemctl start docker</span></code></pre></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><h3 style="text-align: left;"><span style="font-family: inherit; font-size: large;">Verifying Docker Installation</span></h3><div><span style="font-family: inherit; font-size: medium;">You can verify the Docker Engine installation by running hello-word image as shown below</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;"><span class="nb" style="box-sizing: border-box; color: #658b00;">sudo </span>docker run hello-world</span></code></pre></div><div><span style="font-family: inherit; font-size: medium;">This will display welcome image and information on successful installation of Docker engine.</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><h2 style="text-align: left;"><span style="font-family: inherit; font-size: large;">Enabling auto-start for Docker Engine on boot</span></h2><div><span style="font-family: inherit; font-size: medium;">To configure Docker engine and containerd service to start automatically at system boot, run the below commands:<br /><br /></span><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;"><span class="nb" style="box-sizing: border-box; color: #658b00;">sudo </span>systemctl <span class="nb" style="box-sizing: border-box; color: #658b00;">enable </span>docker.service</span></code></pre><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="font-family: inherit; font-size: medium;">sudo <span style="color: #c1f1f0;">systemctl </span><span class="nb" style="box-sizing: border-box; color: #658b00;">enable </span><span style="color: #c1f1f0;">containerd.service</span></span></pre></div><div style="text-align: left;"><span style="font-family: inherit; font-size: medium;"><br /></span><h3 style="text-align: left;"><span style="font-family: inherit; font-size: large;">Changing the data storage directory</span></h3></div><div><span style="font-family: inherit; font-size: medium;">If you need to change the default image installation directory, follow the below steps:</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><span style="font-family: inherit; font-size: medium;">Create or modify the Docker configuration file: <span style="background-color: #404345; color: #e7e8eb; white-space: pre-wrap;">/etc/docker/daemon.json</span> and update/add the <span style="background-color: #404345; color: #e7e8eb; white-space: pre-wrap;">"data-root"</span> value as shown in the example below</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><span style="font-family: inherit; font-size: medium;"><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;">{
<span> "data-root": "/new/path/to/docker-data"
</span>}</pre></span></div><div><span style="font-family: inherit; font-size: medium;">After making the changes, restart the Docker engine service.</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><pre style="background-color: var(--highlight-bg); border-radius: 5px; border: 0px; box-sizing: inherit; color: var(--highlight-color); font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: calc(var(--s-prose-spacing) + 0.4em); margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><code style="background-color: transparent; border-radius: 0px; border: 0px; box-sizing: inherit; color: var(--black-800); font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline; white-space: inherit;"><span style="font-family: inherit; font-size: medium;">sudo systemctl daemon-reload
sudo systemctl restart docker </span></code></pre></pre></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><span style="font-family: inherit; font-size: medium;">Please note, this steps are for new Docker engine configuration, if you are already running docker container images, need to stop the Docker service, change the configuration, move the existing images and start the service to take effect - we are not going to discuss about detailed steps for now.</span></div><div style="text-align: left;"><span style="font-family: inherit; font-size: medium;"><br /></span></div><h2 style="text-align: left;"><span style="font-family: inherit; font-size: large;">Uninstalling Docker Engine (Optional)</span></h2><div><span style="font-family: inherit; font-size: medium;">To uninstall Docker engine, Command Line Interface and Containerd packages run the below command:</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;"><span class="nb" style="box-sizing: border-box; color: #658b00;">sudo </span>yum remove docker-ce docker-ce-cli containerd.io</span></code></pre></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><span style="font-family: inherit; font-size: medium;">Images, containers, volumes and customized configuration files on your machines are not removed automatically, run the below command to do that:</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;"><span class="nb" style="box-sizing: border-box; color: #658b00;">sudo </span>rm <span class="nt" style="box-sizing: border-box; color: #e8a0e8;">-rf</span> /var/lib/docker
<span class="nb" style="box-sizing: border-box; color: #658b00;">sudo </span>rm <span class="nt" style="box-sizing: border-box; color: #e8a0e8;">-rf</span> /var/lib/containerd</span></code></pre></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><span style="font-family: inherit; font-size: medium;">Note, if you have changed the storage path of Docker image installation directory, as seen in previous steps, empty the directory manually to remove the contents.</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><h2 style="text-align: left;"><span style="font-family: inherit; font-size: large;">Installing Docker Compose</span></h2><div><span style="font-family: inherit; font-size: medium;">Docker compose used to defining and running multiple container services required for your application. All application service configurations specified in the <span style="background-color: #0a121b; color: #c1f1f0; white-space: nowrap;"> docker-compose.yml </span> file. With single command you can start and run all the container services from the configuration file.</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><span style="font-family: inherit; font-size: medium;">Docker compose commands provides options manage the entire application lifecycle, such as starting, stopping, rebuilding services, status of the services and streaming running service logs, etc.</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><span style="font-family: inherit; font-size: medium;">Most simplest way to install Docker compose is using <span style="background-color: #0a121b; color: #c1f1f0; white-space: nowrap;"> pip </span> installer (I am extensively using Python, so it is the easiest option for me to manage through virtual environment). Run the below command to install </span><span style="background-color: #0a121b; color: #c1f1f0; font-family: inherit; font-size: large; white-space: nowrap;"> docker-compose </span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><pre class="highlight" style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;">pip install docker-compose</span></code></pre></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><h2 style="text-align: left;"><span style="font-family: inherit;"><span style="font-size: large;">Example docker commands</span></span></h2><div><span style="font-family: inherit; font-size: medium;">Let's try some of the docker command examples:</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><span style="font-size: medium;"><pre style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-family: inherit; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code class="language-none" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="color: #c1f1f0; font-family: inherit; font-size: medium;">docker <command></span></code></pre><pre style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; font-family: inherit; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code class="language-none" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="color: #c1f1f0; font-family: inherit; font-size: medium;">Example: docker-version</span></code></pre><pre style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code class="language-none" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="color: #c1f1f0; font-size: medium;">version Show the Docker version
top Display the running processes
start Start containers
stop Stop containers
stats Display resource usage statistics
service Manage services
rm Remove containers
restart Restart containers
ps List containers
kill Kill running containers
images List images
pull Pull an image or a repository from a registry
push Push an image or a repository to a registry
logs Fetch the logs of a container
build Build an image from a Dockerfile</span><span style="color: #c1f1f0; font-family: inherit; font-size: medium;">
</span></code></pre><div style="font-family: inherit;"><br /></div></span></div><h3 style="text-align: left;"><span style="font-family: inherit; font-size: large;">Example docker-compose commands</span></h3><div><span style="font-family: inherit; font-size: medium;">To get start quickly, let me add few example commands, this will be helpful, when you are dealing with Docker or containerization for the first time.</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><pre style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code class="language-none" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="color: #c1f1f0; font-family: inherit; font-size: medium;">docker-compose <command></span></code></pre><pre style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><span style="font-family: inherit; font-size: medium;"><code class="language-none" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="color: #c1f1f0;">Example: docker-compose up</span></code>start Start services</span></pre></div><div><pre style="background-color: #0a121b; border-radius: 4px; border: 1px solid rgb(0, 0, 0); box-sizing: border-box; color: #f3f3f3; line-height: 1.42857; margin-bottom: 10px; margin-top: 0px; overflow-wrap: break-word; overflow: auto; padding: 20.5px; word-break: normal;"><code class="language-none" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; border-radius: 0px; box-sizing: border-box; color: #c1f1f0; overflow-wrap: normal; overflow-x: auto; padding: 0px;"><span style="font-family: inherit; font-size: medium;"> down Stop and remove containers, networks, images, and volumes
help Get help on a command
images List images
kill Kill containers
logs View output from containers
ps List containers
restart Restart services
rm Remove stopped containers
start Start services
stop Stop services
top Display the running processes
up Create and start containers
version Show the Docker-Compose version information</span></code></pre></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><span style="font-family: inherit; font-size: medium;">I hope this article is helpful to you. Share in comment section, if you are facing any error or challenges in following the above steps.</span></div><div><span style="font-family: inherit; font-size: medium;"><br /></span></div><div><br /></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com0tag:blogger.com,1999:blog-2027709621250502166.post-40596334600597204432021-05-19T11:42:00.010+05:302021-05-20T23:52:45.950+05:30Real-time Monitoring and Log Streaming in Google Cloud Platform (GCP)<p style="text-align: justify;"></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc9-ou0QtnGE0wJuPp-LAV-6aoR7QED_kL-H-ZndlmXrDS6Yc9aqOusi1Jz02AO7ZHqIQ58BvpIYwELPcR6ZAK8CkWPXqCZfmQXgu7CwmojMwqRsi378rQtvhNHXnb0mbjRVTRd_tUkeKR/s1920/monitor.jpeg" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img border="0" data-original-height="1280" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc9-ou0QtnGE0wJuPp-LAV-6aoR7QED_kL-H-ZndlmXrDS6Yc9aqOusi1Jz02AO7ZHqIQ58BvpIYwELPcR6ZAK8CkWPXqCZfmQXgu7CwmojMwqRsi378rQtvhNHXnb0mbjRVTRd_tUkeKR/s16000/monitor.jpeg" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Monitoring Dashboard with charts - credit <a href="https://pixabay.com/" rel="nofollow" target="_blank">pixabay</a> </span></td></tr></tbody></table><span style="font-size: large;"><br /><span style="font-family: Roboto;"><br /></span></span><p></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Getting insights into <b>performance, availability and health status of infrastructure and application</b> is very critical for building and managing reliable systems. When we are dealing with clusters of instances, it is becoming very challenging to <b>collect, aggregate and derive actionable insights from data in real time</b>. There are monitoring tools available to address this challenge, both open source and commercial products, we are going to discuss about how to achieve <b>real-time log streaming analytics and monitoring</b> in <b>Google Cloud Platform</b>.<span></span></span></p><a name='more'></a><p></p><p></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><b>Google Cloud Platform (GCP)</b> provides <b>Cloud Monitoring</b> service to gather <b>system and application statistics</b> and <b>Cloud Logging</b> service to <b>collect and stream logs from system and third-party application</b>. Also, it provides options to enable alerting and integration with incident management tools. To make things interesting let's implement the <b>log streaming and metrics monitoring of Nginx instance</b>. Let me give you quick background of the example application In recent years, I have been predominantly using <b>Google Cloud Platform</b> for hosting <b>Machine Learning</b> <b>workloads</b> and applications. Architecture of the system is not in scope of this article, I will write a separate post on that later, however in simple terms, for computing I have been using <b>Compute Engine Instance Groups</b> consists of Linux Virtual Machines with attached GPUs. <b>Inference engine APIs</b> are hosted on Flask and Gunicorn web server and served through <b>Nginx reverse proxy</b> and traffics are served and managed through Internal HTTPS load balancers and Global Load Balancers. </span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">For gathering system and application metrics and sending to Cloud Monitoring, we need to install <b>Cloud Monitoring Agent</b> on Virtual Machine instance and collecting third-party application logs and streaming to Cloud Logging requires installation of <b>Cloud Logging Agent</b> on Virtual Machine instances. To explore the metrics and create charts to visualize we will use <b>Metric</b> <b>Explorer</b>, and to analyse and explore the logs, we will use <b>Log Explorer</b> interface. In addition to this, we need <b>Nginx monitoring</b> <b>plugin</b> for collecting <b>Nginx application metrics</b>. We will also cover about creating custom <b>monitoring dashboards, widgets, Nginx metrics, Log based metrics, etc.</b></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><h2 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Topics covered:</span></h2><p style="text-align: justify;"></p><ul><li><span style="font-family: Roboto; font-size: large;">Cloud Monitoring Agent Introduction</span></li><ul><li><span style="font-family: Roboto; font-size: large;">Cloud Monitoring Agent installation through SSH</span></li><li><span style="font-family: Roboto; font-size: large;">Cloud Monitoring Agent installation through Google Cloud Console</span></li></ul><li><span style="font-family: Roboto; font-size: large;">Nginx Monitoring plugin Introduction</span></li><ul><li><span style="font-family: Roboto; font-size: large;">Enabling Nginx status page</span></li><li><span style="font-family: Roboto; font-size: large;">Enabling Nginx monitoring plugin</span></li></ul><li><span style="font-family: Roboto; font-size: large;">Cloud Monitoring Dashboard</span></li><ul><li><span style="font-family: Roboto; font-size: large;">Dashboard Widgets</span></li><li><span style="font-family: Roboto; font-size: large;">Nginx Application Dashboard</span></li></ul><li><span style="font-family: Roboto; font-size: large;">Nginx metrics monitored in Google Cloud Monitoring</span></li><li><span style="font-family: Roboto; font-size: large;">Exploring Nginx Metrics in Cloud Monitoring Metrics Explorer</span></li><li><span style="font-family: Roboto; font-size: large;">Cloud Logging Agent Introduction</span></li><ul><li><span style="font-family: Roboto; font-size: large;">Cloud Logging Agent installation through SSH</span></li><li><span style="font-family: Roboto; font-size: large;">Cloud Logging Agent installation through Google Cloud Console</span></li></ul><li><span style="font-family: Roboto; font-size: large;">Enabling Structured Logging in Google Cloud Logging</span></li><li><span style="font-family: Roboto; font-size: large;">Exploring Nginx Logs in Google Cloud Logging - Log Explorer</span></li><ul><li><span style="font-family: Roboto; font-size: large;">Filtering Nginx Logs in Log Explorer</span></li><li><span style="font-family: Roboto; font-size: large;">Real-time streaming of Nginx Logs</span></li></ul><li><span style="font-family: Roboto; font-size: large;">Creating Log based metrics</span></li><li><span style="font-family: Roboto; font-size: large;">Cloud Monitoring Agent – Administrative Tasks</span></li><ul><li><span style="font-family: Roboto; font-size: large;">Restarting and checking status of Cloud Monitoring Agent</span></li><li><span style="font-family: Roboto; font-size: large;">Configuring HTTP proxy for Cloud Monitoring Agent</span></li><li><span style="font-family: Roboto; font-size: large;">Uninstalling Cloud Monitoring Agent</span></li></ul><li><span style="font-family: Roboto; font-size: large;">Cloud Logging Agent – Administrative Tasks</span></li><ul><li><span style="font-family: Roboto; font-size: large;">Restarting and checking status of Cloud Logging Agent</span></li><li><span style="font-family: Roboto; font-size: large;">Configuring HTTP proxy for Cloud Logging Agent</span></li><li><span style="font-family: Roboto; font-size: large;">Uninstalling Cloud Logging Agent</span></li></ul><li><span style="font-family: Roboto; font-size: large;">Summary</span></li><li><span style="font-family: Roboto; font-size: large;">Resources</span></li></ul><h2 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Cloud Monitoring Agent – Introduction:</span></h2><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">The Cloud Monitoring Agent is based on the <b>collectd</b> – the <b>system statistics collection daemon</b>, which <b>gathers system and application metrics from Virtual Machine instances and sends them to monitoring</b>. This system gathers information from various sources, e.g. the <b>operating system, applications, logfiles, and external devices and sends them to Google Cloud Monitoring</b>. These metrics can be visualized through graphs, charts, dashboards and used to monitor systems, find <b>performance bottlenecks and capacity planning, etc</b>. The collectd is written in C for performance and portability, so we can be confident that this will not create additional overhead on the instance. Monitoring agent used to collect CPU, Memory, Disk, Network, system resources and third-party applications</span></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Installation:</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">The Cloud Monitoring Agent can be installed in below methods</span></p><p style="text-align: justify;"></p><ul><li><span style="font-family: Roboto; font-size: large;">Installing through SSH</span></li><li><span style="font-family: Roboto; font-size: large;">Installing through Google Cloud Console – installation workflow</span></li></ul><p></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Cloud Monitoring Installation through SSH:</span></h3><p style="text-align: justify;"></p><ul><li><span style="font-family: Roboto; font-size: large;">SSH into your Virtual Machine instance and create a directory monitoring</span></li><li><span style="font-family: Roboto; font-size: large;">Change directory to monitoring and download the Logging Agent installation script</span></li></ul><p></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>curl -sSO https://dl.google.com/cloudagents/add-monitoring-agent-repo.sh</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"></p><ul><li><span style="font-family: Roboto; font-size: large;">Run the below command to add the repository and install the agent in your machine</span></li></ul><p></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><div></div><span style="font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><div style="text-align: left;"><span style="font-size: large;"><span> </span>sudo bash add-monitoring-agent-repo.sh --also-install</span></div><div style="text-align: left;"><span style="font-size: large;"><br /></span></div></div><p></p><p></p><ul style="text-align: left;"><li><span style="font-family: Roboto; font-size: large;">To verify the status of the Google Cloud Logging agent, run the below command</span></li></ul><p></p><p></p><p></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span> </span>sudo systemctl status stackdriver-agent</span></div><div><span style="font-size: large;"><br /></span></div></div><p></p><ul style="text-align: left;"><li><span style="font-family: Roboto; font-size: large;">Run the below command to add the repository and install the agent in your machine</span></li></ul><p></p><p></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span> </span>sudo grep collectd /var/log/{syslog,messages} | tail</span></div><div><span style="font-size: large;"><br /></span></div></div><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></h3><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Cloud Monitoring Installation through Google Cloud Console:</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Log into Google Cloud Console and navigate to: <b>Google Cloud Console Menu -> Monitoring -> Dashboards -> VM Instances -> Inventory</b>. This page will provide about current status of Monitoring Agent. </span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIcn4aFFaQHtzfhj6ICcsQAmDCYJmgAj3cF_xjrq_P8JwbwB4mKqoCZAw9vzMBTKUcythyPzzhfnZQMMPKlhGhjKCxwLWYUtOUS3tBJT54loD6xhg6WWfaheNJrJn8fi5rykFTK5er4WfH/s2878/VM+Instances+Dashboard.png" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Google Cloud Monitoring - VM Instances - Inventory" border="0" data-original-height="922" data-original-width="2878" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIcn4aFFaQHtzfhj6ICcsQAmDCYJmgAj3cF_xjrq_P8JwbwB4mKqoCZAw9vzMBTKUcythyPzzhfnZQMMPKlhGhjKCxwLWYUtOUS3tBJT54loD6xhg6WWfaheNJrJn8fi5rykFTK5er4WfH/s16000/VM+Instances+Dashboard.png" title="Google Cloud Monitoring - VM Instances - Inventory" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Google Cloud Monitoring - VM Instances Inventory</span></td></tr></tbody></table><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">Click on the <b>Install Agent</b> button - it will open the agent details page with installation workflow, once you click on Install Agent, <b>Google Cloud Shell</b> will open with installation command, press Enter to complete the installation.</span></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHJriMOxs3RYKWnAG7DnDqwOfktZhuROfj4TD_AcmOdsOrCKvh5i_GMKCuYeUi-apggf7Vb2_BRUX4U7VRIcIdIVHiqhcSrTuDB7q4stDpONnGKHXPc2Lq880v4Mqxxe-sxOZ67hpzsv_m/s2048/Monitoring+Agent+Details.png" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Google Cloud Monitoring Agent - Installing through Google Cloud Console" border="0" data-original-height="1051" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHJriMOxs3RYKWnAG7DnDqwOfktZhuROfj4TD_AcmOdsOrCKvh5i_GMKCuYeUi-apggf7Vb2_BRUX4U7VRIcIdIVHiqhcSrTuDB7q4stDpONnGKHXPc2Lq880v4Mqxxe-sxOZ67hpzsv_m/s16000/Monitoring+Agent+Details.png" title="Google Cloud Monitoring Agent - Installing through Google Cloud Console" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Google Cloud Monitoring - Agent Installation Details</span></td></tr></tbody></table><p></p><h2><span style="font-family: Roboto; font-size: large;"><br /></span></h2><h2><span style="font-family: Roboto; font-size: large;">Nginx Monitoring plugin:</span></h2><p></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">We have successfully installed the Google Cloud Monitoring Agent, now let’s install and configure the <b>Nginx monitoring plugin</b>. By default the Google Cloud Monitoring Agent can discover the <b>Nginx logs based on configuration file and stream to Cloud Monitoring</b> and additionally Google Cloud Logging Agent also streams logs to Logging Explorer, however we are proceeding with installing Nginx monitoring plugin to create a <b>Nginx Application Dashboard</b> where all Nginx instances can be monitored at a time.</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Nginx monitoring installation involves 2 steps, configuring <b>nginx_status</b> handler to enable nginx status page and enabling <b>Nginx monitoring plugin</b>.</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><h4 style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">Enabling and configuring Nginx status page</span></span></h4><p style="text-align: justify;"></p><ul><li><span style="font-family: Roboto; font-size: large;">Create a new configuration file status.conf inside your nginx configuration directory <b>/etc/nginx/conf.d/</b></span></li></ul><p></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span style="color: #56b6c2;"><span> </span>cd</span> /etc/nginx/conf.d/</span></div><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span> </span>vi status.conf</span></div><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"></p><ul><li><span style="font-family: Roboto; font-size: large;">Add the below directives inside <b>status.conf</b> file – or you can download this file from <a href="https://github.com/Stackdriver/stackdriver-agent-service-configs/blob/master/etc/nginx/conf.d/status.conf">https://github.com/Stackdriver/stackdriver-agent-service-configs/blob/master/etc/nginx/conf.d/status.conf</a> and save it in configuration directory.</span></li></ul><p></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><div><span style="font-size: large;"><br /></span></div><div><div style="line-height: 20px;"><div><span style="font-size: large;"> server {</span></div><div><span style="font-size: large;"> listen 80;</span></div><div><span style="font-size: large;"> server_name local-stackdriver-agent.stackdriver.com;</span></div><div><span style="font-size: large;"> location /nginx_status {</span></div><div><span style="font-size: large;"> stub_status on;</span></div><div><span style="font-size: large;"> access_log off;</span></div><div><span style="font-size: large;"> allow 127.0.0.1;</span></div><div><span style="font-size: large;"> deny all;</span></div><div><span style="font-size: large;"> }</span></div><div><span style="font-size: large;"> location / {</span></div><div><span style="font-size: large;"> root /dev/null;</span></div><div><span style="font-size: large;"> }</span></div><div><span style="font-size: large;"> }</span></div></div></div><div><span style="font-size: large;"><br /></span></div></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"></p><ul><li><span style="font-family: Roboto; font-size: large;">Alternatively, you can append the above code block in your Nginx configuration file – generally located at <b>/etc/nginx/nginx.conf</b></span></li><li><span style="font-family: Roboto; font-size: large;">Reload the Nginx configuration by running:</span></li></ul><p></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>sudo service nginx reload</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><h4 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></h4><h4 style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">Enabling Nginx monitoring plugin:</span></span></h4><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Download the nginx.conf file from <a href="https://github.com/Stackdriver/stackdriver-agent-service-configs/blob/master/etc/collectd.d/nginx.conf">https://github.com/Stackdriver/stackdriver-agent-service-configs/blob/master/etc/collectd.d/nginx.conf</a> and save it place it under the Cloud Monitoring Agent location <b>/etc/stackdriver/collectd.d/</b></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Restart the Cloud Monitoring Agent by running the below command:</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>sudo systemctl restart stackdriver-agent</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;"><br /></span></span></p><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">To check the Cloud Monitoring Agent status:</span></span></p><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;"><br /></span></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span> </span>sudo systemctl status stackdriver-agent</span></div><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><h2 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Cloud Monitoring Dashboard:</span></h2><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Monitoring dashboard is one of the ways to <b>view and analyse your metrics</b> in Google Cloud Platform. Google Cloud Platform provides dashboards for most of the Google Cloud Services without any effort from our end. It also provide options to <b>monitor custom metrics thorough custom dashboard</b>. We have option to add <b>Line Chart, Stacked area chart, Stacked bar chart, Heatmap chart, Gauges, Scorecards and textboxes widgets</b> to our custom dashboard as required. </span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Nginx Application Dashboard</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">As we have already added, Nginx monitoring plugin, we should be able to see <b>Nginx Application dashboard</b> in our Monitoring Dashboard page. Navigate to <b>Google Cloud Console -> Monitoring -> Dashboard to see Nginx dashboard</b>:</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"> </span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtbzzSW9k3IEGiOL7HTYh1j3XE6cOwjO0h9qsgxDsm8pg7nScV5EiiSdNMRQTrMdPgMGavZQVlEATtHN2DVoEqYSShdUH1cqFsRwrdqxUxKIWXR3hZw9zpmoF1BYxaT7Bp-uasWeB3LeKv/s2598/Nginx+dashboard.png" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Google Cloud Monitoring - Nginx Application Dashboard" border="0" data-original-height="1210" data-original-width="2598" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtbzzSW9k3IEGiOL7HTYh1j3XE6cOwjO0h9qsgxDsm8pg7nScV5EiiSdNMRQTrMdPgMGavZQVlEATtHN2DVoEqYSShdUH1cqFsRwrdqxUxKIWXR3hZw9zpmoF1BYxaT7Bp-uasWeB3LeKv/s16000/Nginx+dashboard.png" title="Google Cloud Monitoring - Nginx Application Dashboard" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Google Cloud Monitoring - Nginx Application Dashboard</span></td></tr></tbody></table><span style="font-family: Roboto; font-size: large;"><br /></span><p></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Click on <b>Nginx Application dashboard</b> (you can use Filter Dashboard to navigate quickly, if you have many dashboards) to see Nginx monitored metrics.</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhc6X9sOMj1IDCn386AeQprlY64Tx38KC8AfPchY05LEajRzFWkKN-VQu7E61Vm9R0kEPokkhjdXnyGZfeW43M9dAPSpHlOOeM1VnI0O4z5Spiqns_iSic11Zv85_Ir9fxh8Htzz9T08bo_/s2528/Nginx+Metrics.png" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Cloud Monitoring Dashboard - Nginx Metrics Dashboard" border="0" data-original-height="1244" data-original-width="2528" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhc6X9sOMj1IDCn386AeQprlY64Tx38KC8AfPchY05LEajRzFWkKN-VQu7E61Vm9R0kEPokkhjdXnyGZfeW43M9dAPSpHlOOeM1VnI0O4z5Spiqns_iSic11Zv85_Ir9fxh8Htzz9T08bo_/s16000/Nginx+Metrics.png" title="Cloud Monitoring Dashboard - Nginx Metrics Dashboard" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Cloud Monitoring Dashboard - Nginx Metrics Dashboard</span></td></tr></tbody></table><span style="font-family: Roboto; font-size: large;"><br /></span><p></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Nginx Metrics monitored in Google Cloud Monitoring:</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">By default, Nginx monitoring plugin collects following metrics:</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"></span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqUp0bjCS9rbnrx1Sx5DYYDWy4CsxbearLkez2OlHHU8MGtq8qSuMwC3odWc2OXzeTPyXgga2wC8MNynzLCuwDjMDiS4n7kOvMRHMh_2iov2rPMDk5XweqmpZdAxFObfP-I0zJ8DlBg0CT/" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Google Cloud Monitoring - Nginx Monitoring Plugin Metrics" data-original-height="69" data-original-width="498" height="88" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqUp0bjCS9rbnrx1Sx5DYYDWy4CsxbearLkez2OlHHU8MGtq8qSuMwC3odWc2OXzeTPyXgga2wC8MNynzLCuwDjMDiS4n7kOvMRHMh_2iov2rPMDk5XweqmpZdAxFObfP-I0zJ8DlBg0CT/w640-h88/image.png" title="Google Cloud Monitoring - Nginx Monitoring Plugin Metrics" width="640" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: large;">Google Cloud Monitoring - Nginx Monitoring Plugin Metrics</span></td></tr></tbody></table><p></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Additionally, you can navigate to metrics explorer from Nginx dashboard by clicking on <b>View in Metrics Explorer</b></span></p><p style="text-align: justify;"></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_mzKaJEGXRPJ2ymCavvks8liFCXJnF9oQqSsh-4jjHRSB_ERTux3OUa4mfjmI-NsW6DfkwXZUQzzXQ2VtsnOtnENMjHPudtLNDh8S6HVlPLwxPsacIQ9v2AKGaIljL9Mm4Oo7wn1uJw4L/s1710/Chart+-%253E+View+in+Metrics+Explorer.png" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Google Cloud Monitoring - Charts View in Metrics Explorer" border="0" data-original-height="936" data-original-width="1710" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_mzKaJEGXRPJ2ymCavvks8liFCXJnF9oQqSsh-4jjHRSB_ERTux3OUa4mfjmI-NsW6DfkwXZUQzzXQ2VtsnOtnENMjHPudtLNDh8S6HVlPLwxPsacIQ9v2AKGaIljL9Mm4Oo7wn1uJw4L/s16000/Chart+-%253E+View+in+Metrics+Explorer.png" title="Google Cloud Monitoring - Charts View in Metrics Explorer" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Google Cloud Monitoring - Charts View in Metrics Explorer</span></td></tr></tbody></table><p></p><div class="separator" style="clear: both; text-align: center;"><span style="font-family: Roboto; font-size: large;"><br /></span></div><p></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Exploring Nginx Metrics in Cloud Monitoring Metrics Explorer</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><b>Metric Explorer lets you explore data and build charts from the monitored metrics</b>. Navigate to metrics explorer through <b>Google Cloud Console -> Monitoring -> Metrics Explorer</b>:</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"></span></p><h2 style="text-align: justify;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoP8GAuPAejxiHTmIv9cERR_PyiPC9jvExU8TG4tYSapaqptcIfwsy6-fekaYulfW7afXVcIHWVMqTfuowzLTIqPzyfyEP_S9u-Hv-g0pppMWzMNELmEIR6hpanJFBAr9r2O04qRrTO1kF/s2575/Nginx+Metrics+Explorer.png" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Google Cloud Monitoring - Exploring Nginx Metrics in Cloud Monitoring Metrics Explorer" border="0" data-original-height="1222" data-original-width="2575" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoP8GAuPAejxiHTmIv9cERR_PyiPC9jvExU8TG4tYSapaqptcIfwsy6-fekaYulfW7afXVcIHWVMqTfuowzLTIqPzyfyEP_S9u-Hv-g0pppMWzMNELmEIR6hpanJFBAr9r2O04qRrTO1kF/s16000/Nginx+Metrics+Explorer.png" title="Google Cloud Monitoring -Exploring Nginx Metrics in Cloud Monitoring Metrics Explorer" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: large;"><br /></span></td></tr></tbody></table></h2><h2 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Google Cloud Logging Agent</span></h2><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><b>Google Cloud Logging Agent</b> is an application based on <b>Fluentd</b> – an open source data collector for unified logging layer. The Logging agent streams data from <b>common-third party applications such as Nginx, MySQL, PostgreSQL, etc</b> and system software to Google Cloud Logging. Also provides configuration option to add custom logs. This architecture allows to <b>unify log data collections and consumption for a better understanding of data and leads to actionable insights.</b> Google Cloud Platform suggests to install the Logging Agent to all the VM instances and it supports both Linux and Windows.</span></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></h3><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Installing Cloud Logging Agent:</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Cloud Logging Agent can be installed in below methods:</span></p><p style="text-align: justify;"></p><ul><li><span style="font-family: Roboto; font-size: large;">Installing Cloud Logging Agent through SSH</span></li><li><span style="font-family: Roboto; font-size: large;">Installing Cloud Logging Agent through Google Cloud Console.</span></li></ul><p></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Cloud Logging Agent installation through SSH</span></h3><p style="text-align: justify;"></p><p style="text-align: left;"></p><ul><li><span style="font-family: Roboto; font-size: large;">SSH into your Virtual Machine instance and create a directory monitoring</span></li></ul><ul><li><span style="font-family: Roboto; font-size: large;">Change directory to monitoring and download the Logging Agent installation script</span></li></ul><p></p><div></div><p></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span> </span>curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh</span></div><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"></p><p></p><p></p><ul style="text-align: left;"><li><span style="font-family: Roboto;"><span style="font-size: large;">Run the below command to add the repository and install the agent in your machine</span></span></li></ul><p></p><p></p><p></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span> </span>sudo bash add-logging-agent-repo.sh --also-install</span></div><div><span style="font-size: large;"><br /></span></div></div><p style="text-align: justify;"></p><p></p><p></p><ul style="text-align: left;"><li><span style="font-family: Roboto; font-size: large;">To verify the status of the Google Cloud Logging agent, run the below command</span></li></ul><p></p><p></p><p></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>sudo systemctl status google-fluentd</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"> </span></div><p style="text-align: justify;"></p><p></p><p></p><ul style="text-align: left;"><li><span style="font-family: Roboto; font-size: large;">If any errors, you can verify the system logs for troubleshooting</span></li></ul><p></p><p></p><p></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>tail /var/log/google-fluentd/google-fluentd.log</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Cloud Logging Agent installation through Google Cloud Console:</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">You can install the Logging Agent from <b>Google Cloud Monitoring</b> page. For this, log into Google Cloud Console and navigate to below page:</span></p><p style="text-align: justify;"><span style="font-size: large;"><span style="font-family: Roboto;"><b>Google Cloud Console Menu -> Monitoring -> Dashboards -> VM Instances -> Inventory. </b></span><span style="font-family: Roboto;">This page will provide information about current status of Logging Agent.</span></span></p><p style="text-align: justify;"></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEganZtp1ZEgsncboXjw05o2SiNCqpLnHNi_pY-iOLxqvH5WiPr7FlnswxoaHeCgpVGdjiOsWG2Iw7JR7002H6xhcgzfK4V7jBSBa2YKIuwx9Gr7KRaRZT4ZJ26-5EcV4X1s28Thu01pLIyP/s2878/VM+Instances+Dashboard.png" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Google Cloud Logging - installing Cloud Logging Agent through Cloud Console" border="0" data-original-height="922" data-original-width="2878" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEganZtp1ZEgsncboXjw05o2SiNCqpLnHNi_pY-iOLxqvH5WiPr7FlnswxoaHeCgpVGdjiOsWG2Iw7JR7002H6xhcgzfK4V7jBSBa2YKIuwx9Gr7KRaRZT4ZJ26-5EcV4X1s28Thu01pLIyP/s16000/VM+Instances+Dashboard.png" title="Google Cloud Logging - installing Cloud Logging Agent through Cloud Console" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Google Cloud Logging - installing Cloud Logging Agent through Cloud Console</span></td></tr></tbody></table><span style="font-family: Roboto; font-size: large;"> </span><p></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">Click on the Install Agent button – it will open the installation workflow on the same page and click install to continue.</span></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6LBMG8s2BLvo2IPnITLTiubZmluv6MDIQ8tLeTXY-G3YRy2EvyOk82oZuktQH9DHRlbKjVOk6musEK9rh68qd-npa2Ve40kLH6M7KqyrO2VGIJ-1YGhZ4uPtg-jsOlrrgT0gIRaWlv1YY/s2048/2+Agent+Details.png" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Google Cloud Logging - Cloud Logging Agent Installation" border="0" data-original-height="1049" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6LBMG8s2BLvo2IPnITLTiubZmluv6MDIQ8tLeTXY-G3YRy2EvyOk82oZuktQH9DHRlbKjVOk6musEK9rh68qd-npa2Ve40kLH6M7KqyrO2VGIJ-1YGhZ4uPtg-jsOlrrgT0gIRaWlv1YY/s16000/2+Agent+Details.png" title="Google Cloud Logging - Cloud Logging Agent Installation" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Cloud Logging Agent Installation</span></td></tr></tbody></table><span style="font-family: Roboto; font-size: large;"><br /></span><p></p><h3 style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">Enabling Structured Logging:</span></span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">We have successfully installed the <b>Google Cloud Logging Agent</b>, now let’s enable the structured logging. In Google cloud logging,<b> structured logs refers to log entries that use jsonPayload field to add structure to their payloads.</b> Structured logs are easy to read and optimal for filtering, this is super helpful when you are filtering the log records later.</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">To enable structure logging, run the below command, this will uninstall the default google-fluentd-catch-all-config packages and install the google-fluentd-catch-all-config-structured</span></p><p style="text-align: justify;"><span style="font-size: large;"><span style="font-family: Roboto;"><br /></span><span style="font-family: Roboto;">Uninstalling Cloud Logging Agent:</span></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>sudo dnf remove google-fluentd-catch-all-config</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-size: large;"><span style="font-family: Roboto;">I</span><span style="font-family: Roboto;">nstalling Cloud Logging Agent - Structured Logging packages:</span></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>sudo dnf install -y google-fluentd-catch-all-config-structured </span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Restarting Cloud Logging Agent:</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>systemctl restart google-fluentd</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Check status of Cloud Logging Agent:'</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"> </span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>systemctl status google-fluentd</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Exploring Nginx Logs in Google Cloud Logging - Log Explorer</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Google Cloud Platform provides <b>Log Explorer</b> interface to retrieve, view and analyse logs from VM instances. We have already configured Cloud Logging Agent on our Nginx VM instance, we can directly query Nginx logs from Log Explorer.</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"> </span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Navigate to Log Explorer <b>Google Cloud Console -> Logging -> Log Explorer</b> to query Nginx logs.</span></p><p style="text-align: justify;"></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimRBD_4BKQueIDvKTx5vGf31P62t_bcCWhMGfADLEYt5PoDYX2rQlUhuNmPIA0xwSOn7rNJtNLS6_IIiMo7U5hVksir2OeRIdofjSilBwhd78e1ymazf-FxlRzncJCjDzLgr37hy5-qqYs/s1816/Log+Explorer.png" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Viewing Nginx Logs in Google Cloud Log Explorer" border="0" data-original-height="897" data-original-width="1816" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimRBD_4BKQueIDvKTx5vGf31P62t_bcCWhMGfADLEYt5PoDYX2rQlUhuNmPIA0xwSOn7rNJtNLS6_IIiMo7U5hVksir2OeRIdofjSilBwhd78e1ymazf-FxlRzncJCjDzLgr37hy5-qqYs/s16000/Log+Explorer.png" title="Viewing Nginx Logs in Google Cloud Log Explorer" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Viewing Nginx Logs in Google Cloud Log Explorer</span></td></tr></tbody></table><span style="font-family: Roboto; font-size: large;"><br /></span><p></p><h3 style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">Filtering Nginx Logs in Log Explorer:</span></span></h3><p style="text-align: justify;"></p><ul><li><span style="font-family: Roboto; font-size: large;"><b>Resource</b> drop down - provides list of all Google Cloud services and Cloud Logging Agent Resources select the Nginx VM instances names</span></li><li><span style="font-family: Roboto; font-size: large;"><b>Log name</b> drop down - provides list of all streamed logs – select the nginx-access and nginx-error logs to get the Nginx Access log and Nginx Error log.</span></li><li><span style="font-family: Roboto; font-size: large;"><b>Severity - </b>leave default to get all severity levels such as Default, Info, Warning, Error, Critical, etc.</span></li></ul><p></p><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">You can write your own queries and apply filtering, also you can save those queries for later stream. Sample Nginx Access Log filtering given below:</span></span></p><p style="text-align: justify;"></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjojKjW2vucUd66m_pDuRGUsBkfuM9J1cpFFahNE2dx7wdYJglxosBPlXUNotKvdh92QTcLsHvPqKK33vkUuIYMNDevxJfpyHgbIBh-gktRskBH1DUfpWAJvgUy0S7D2zNe-fOrFONVYU6T/s1720/Log+Explorer+Filtering.png" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Filtering Nginx Logs in Google Cloud Logging - Log Explorer" border="0" data-original-height="1720" data-original-width="1376" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjojKjW2vucUd66m_pDuRGUsBkfuM9J1cpFFahNE2dx7wdYJglxosBPlXUNotKvdh92QTcLsHvPqKK33vkUuIYMNDevxJfpyHgbIBh-gktRskBH1DUfpWAJvgUy0S7D2zNe-fOrFONVYU6T/s16000/Log+Explorer+Filtering.png" title="Filtering Nginx Logs in Google Cloud Logging - Log Explorer" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Filtering Nginx Logs in Google Cloud Logging - Log Explorer</span></td></tr></tbody></table><span style="font-size: large;"><br /></span><p></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Real-time streaming of Nginx Logs:</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Log Explorer provides option to stream the logs real-time, this is super useful, when you are troubleshooting issues and all filters will be applied on the log stream as well. Click on the Start stream button to stream logs, once done, you can use Stop stream button to stop the streaming</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoAdgdzO68NSilftkXfmSsyutOQMK7mMmNH5NMu_azDSJopUT_hvUSXWJs0z25BOaG5k6tU9Dxt_e0jBXZT03Idmx30a6RW_CMkSe_EAMce3TWTIJ4b1HNRljfHzWkWPYVU499MXt5P6yr/s2921/Real-time+Streaming.png" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Real-time streaming of Nginx Logs - Google Cloud Logging - Log Explorer stream" border="0" data-original-height="1077" data-original-width="2921" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoAdgdzO68NSilftkXfmSsyutOQMK7mMmNH5NMu_azDSJopUT_hvUSXWJs0z25BOaG5k6tU9Dxt_e0jBXZT03Idmx30a6RW_CMkSe_EAMce3TWTIJ4b1HNRljfHzWkWPYVU499MXt5P6yr/s16000/Real-time+Streaming.png" title="Real-time streaming of Nginx Logs - Google Cloud Logging - Log Explorer stream" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Real-time streaming of Nginx Logs - Google Cloud Logging - Log Explorer stream</span></td></tr></tbody></table><span style="font-size: large;"><br /></span><p></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Creating Log based metrics:</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Log based metrics are useful when you need to monitor a metric based on the content of file. For example, consider you need to measure the total number of POST request served by the Nginx. A Log based metric can be either a <b>Counter type</b> or <b>Distribution type</b>. Our example is type of Counter metric – count the number of log entries matching a given filter and Distribution metric accumulates numeric data from log entries matching a filter.</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">To create Log based metric click on <b>Actions -> Create Metrics</b> link. This will open Log based metrics page, where you can config a new metrics based on the log content.</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"></span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4Mf-nugDcn9PSZV4_y-Fl8a_xOWdFGOa5sCm12x7HJ6KO5Q26jOodIU-CpRRGwart_2_YnqXIvfW-Fm_OZkp-_sJ0Mb3rR_DvJStEOeNFkwUpLuUzW4xWt32ejlIXEBclAUpw7SA_V3FU/s1786/Log+based+metrics.png" style="margin-left: auto; margin-right: auto;"><span style="font-size: large;"><img alt="Google Cloud Logging - Creating Log Based metrics" border="0" data-original-height="1786" data-original-width="938" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4Mf-nugDcn9PSZV4_y-Fl8a_xOWdFGOa5sCm12x7HJ6KO5Q26jOodIU-CpRRGwart_2_YnqXIvfW-Fm_OZkp-_sJ0Mb3rR_DvJStEOeNFkwUpLuUzW4xWt32ejlIXEBclAUpw7SA_V3FU/s16000/Log+based+metrics.png" title="Google Cloud Logging - Creating Log Based metrics" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: x-small;">Google Cloud Logging - Creating Log Based metrics</span></td></tr></tbody></table><span style="font-family: Roboto; font-size: large;"><br /> </span><p></p><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">Also, you can use this <b>Log based metrics in Metrics Explorer</b> and create visualization and can be added to your custom dashboards.</span></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><h2 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Cloud Monitoring Agent – Administrative Tasks</span></h2><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">In this section, we will discuss about how to restart and check status of the Cloud Monitoring Agent, configuring HTTP proxy for Cloud Monitoring Agent and Uninstalling Cloud Monitoring Agent.</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><h3 style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">Restarting & checking status of Cloud Monitoring Agent:</span></span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Restart:</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>sudo systemctl restart stackdriver-agent</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">Status:</span></span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"> </span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>sudo systemctl status stackdriver-agent</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-size: large;"><br /></span></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Configuring HTTP Proxy:</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Edit the Cloud Monitoring Agent configuration file:</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span> </span>/etc/default/stackdriver-agent</span></div><div><span style="font-size: large;"><br /></span></div></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Add the proxy details:</span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span style="color: #c678dd;"><span> </span>export</span> http_proxy=<span style="color: #98c379;">"http://proxy-ip:proxy-port"</span></span></div><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span style="color: #c678dd;"><span> </span>export</span> https_proxy=<span style="color: #98c379;">"http://proxy-ip:proxy-port"</span></span></div><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span style="color: #c678dd;"><span> </span>export</span> no_proxy=169.254.169.254 <span style="color: #7f848e; font-style: italic;"># Skip proxy for the local Metadata Server.</span></span></div><div><span style="font-size: large;"><br /></span></div></div><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Restart the Cloud Monitoring Agent:</span></p><div><span style="font-family: Roboto; font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"> </span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>sudo systemctl restart stackdriver-agent</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-size: large;"><br /></span></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Uninstalling Cloud Monitoring Agent:</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Run the below command to uninstall the Cloud Monitoring Agent:</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"> </span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>sudo bash add-monitoring-agent-repo.sh –uninstall</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-size: large;"><br /></span></p><h2 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Cloud Logging Agent – Administrative Tasks</span></h2><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">In this section, we will discuss about how to restart and check status of the Cloud Logging Agent, configuring HTTP proxy for Cloud Logging Agent and Uninstalling Cloud Logging Agent.</span></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></h3><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Restarting & checking status of Cloud Logging Agent:</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Restart:</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><div><span style="font-size: large;"><br /></span></div><div><span style="font-size: large;"><span> </span>sudo systemctl restart google-fluentd </span></div><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">Status:</span></span></p><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;"><br /></span></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"> </span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>sudo systemctl status google-fluentd</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Configuring HTTP Proxy:</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Edit the Cloud Logging Agent configuration file:</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><div><span style="font-size: large;"> </span></div><div><span style="font-size: large;"><span> </span>/etc/default/google-fluentd</span></div><div><span style="font-size: large;"><br /></span></div></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">Add the proxy details:</span></span></p><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;"><br /></span></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><div><span style="color: #c678dd;"><span style="font-size: large;"> </span></span></div><div><span style="font-size: large;"><span style="color: #c678dd;"><span> </span>export</span> http_proxy=<span style="color: #98c379;">"http://proxy-ip:proxy-port"</span></span></div><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span style="color: #c678dd;"><span> </span>export</span> https_proxy=<span style="color: #98c379;">"http://proxy-ip:proxy-port"</span></span></div><span style="font-size: large;"><br /></span><div><span style="font-size: large;"><span style="color: #c678dd;"><span> </span>export</span> no_proxy=169.254.169.254 <span style="color: #7f848e; font-style: italic;"># Skip proxy for the local Metadata Server.</span></span></div><div><span style="color: #7f848e; font-size: large; font-style: italic;"><br /></span></div></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;">Restart the Cloud Monitoring Agent:</span></span></p><p style="text-align: justify;"><span style="font-family: Roboto;"><span style="font-size: large;"><br /></span></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>sudo systemctl restart google-fluentd</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Uninstalling Cloud Monitoring Agent:</span></h3><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Run the below command to uninstall the Cloud Monitoring Agent:</span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><span> </span>sudo bash add-logging-agent-repo.sh –uninstall</span></div><div style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; line-height: 20px; white-space: pre;"><span style="font-size: large;"><br /></span></div><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><h2 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Summary</span></h2><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">We have covered the following topics in this article and I hope this will be helpful in implementing monitoring solution for your application. Share your comments and queries below in the comment section.</span></p><p style="text-align: justify;"></p><ul><li><span style="font-family: Roboto; font-size: large;"><b>Cloud Monitoring Agen</b>t and installing, configuring Cloud Monitoring Agent on <b>Linux Virtual Machines</b></span></li><li><span style="font-family: Roboto; font-size: large;"><b>Cloud Logging Agent</b> and installing, configuring Cloud Logging Agent on <b>Linux Virtual Machines</b></span></li><li><span style="font-family: Roboto; font-size: large;"><b>Nginx monitoring Plugin</b> and enabling <b>Nginx status page</b> and <b>enabling Nginx monitoring</b> from Cloud Monitoring Agent</span></li><li><span style="font-family: Roboto; font-size: large;"><b>Monitoring dashboards</b>, different types of widgets and <b>Nginx Application Dashboard</b>, exploring Nginx metrics in <b>Cloud Monitoring Metrics Explorer</b> interface</span></li><li><span style="font-family: Roboto; font-size: large;"><b>Viewing and analysing log data</b> in <b>Cloud Logging Log Explorer</b> interface and real-time streaming logs through Log Explorer and creating <b>Log based metrics</b></span></li><li><span style="font-family: Roboto; font-size: large;"><b>Cloud Monitoring Agent and Logging Agent </b>administrative tasks such as <b>restarting, checking status, configuring proxy and uninstallation</b>.</span></li></ul><p></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><h2 style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Additional Resources</span></h2><p style="text-align: justify;"><span style="font-size: large;"><span style="font-family: Roboto;">Google Cloud Console: <a href="https://console.cloud.google.com/">https://console.cloud.google.com/</a></span><span style="font-family: Roboto;"><a href="https://console.cloud.google.com/ " rel="nofollow" target="_blank"> </a></span></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Nginx Logging: <a href="https://docs.nginx.com/nginx/admin-guide/monitoring/logging/">https://docs.nginx.com/nginx/admin-guide/monitoring/logging/</a> </span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Cloud Monitoring Agent: <a href="https://cloud.google.com/monitoring/agent/monitoring">https://cloud.google.com/monitoring/agent/monitoring</a> </span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Cloud Logging Agent: <a href="https://cloud.google.com/logging/docs/agent/logging">https://cloud.google.com/logging/docs/agent/logging</a></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">Fluentd Architecture: <a href="https://www.fluentd.org/architecture " rel="nofollow" target="_blank">https://www.fluentd.org/architecture </a></span></p><p style="text-align: justify;"><span style="font-family: Roboto; font-size: large;">collectd: <a href="https://collectd.org/index.shtml " rel="nofollow" target="_blank">https://collectd.org/index.shtml </a></span></p>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com0tag:blogger.com,1999:blog-2027709621250502166.post-44680361541782339862021-05-18T11:38:00.001+05:302021-05-19T11:44:43.874+05:30Understanding MySQL Architecture<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="MsoNormal" style="text-align: justify;">
The architecture of the world’s
most popular open source database system is very important for the Information
Technology people. There are many reasons for MySQL’s popularity around the
world, but one of the main reasons is its architecture, while there are many
big players such as Oracle, Microsoft SQL and DB2, MySQL’s architecture makes
it as unique and preferred choice for most of the developers. In this article,
we are going to discuss about of the internal architecture of the MySQL
relational database management system. The article is for novice database
administrators, database developers, software developers and those who are
interested to work with MySQL database.<span></span><br /><o:p></o:p></div>
<h2 style="text-align: justify;">
<br /><b>Major components:</b></h2>
<div>
<div class="MsoNormal" style="text-align: justify;">
The MySQL architecture describes
how the different components of a MySQL system relate to one another. The MySQL
architecture is basically a client – server system. MySQL database server is
the server and the applications which are connecting to MySQL database server
are clients. The MySQL architecture contains the following major components.<o:p></o:p></div>
</div>
<div>
<b><br /></b></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlGKJNTOZSW2qyEbr4U8a5NF1hoQPaZkzKgsBqHS8Lp51jBgPdpSQNX55OX98TO8l99m6Wruqz2itjQRS5cefVsnJXzQpuzlOeNcwhXdTvV8l_3XLCzIzbGwNz7XhW1B2-S9Sh9qPxijPp/s1600/MySQL+Architecture.png" style="margin-left: auto; margin-right: auto;"><img alt="MySQL Architecture" border="0" height="335" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlGKJNTOZSW2qyEbr4U8a5NF1hoQPaZkzKgsBqHS8Lp51jBgPdpSQNX55OX98TO8l99m6Wruqz2itjQRS5cefVsnJXzQpuzlOeNcwhXdTvV8l_3XLCzIzbGwNz7XhW1B2-S9Sh9qPxijPp/s400/MySQL+Architecture.png" title="MySQL Architecture" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">MySQL ARCHITECTURE</td></tr>
</tbody></table>
<br />
<h2 style="text-align: justify;">
<b><span><a name='more'></a></span>Application Layer:</b></h2>
<div class="MsoNormal" style="text-align: justify;">
This layer is the top most layers
in MySQL architecture; you can see this same layer in many of the client –
server architecture. This layer includes some of the services which are common
to most of the client – server applications, some of the services are given
below:<br />
<div class="MsoNormal">
</div>
<ul>
<li>Connection Handling.</li>
<li>Authentication.</li>
<li>Security.</li>
</ul>
<br />
<h2>
<b>Connection Handling:</b></h2>
</div>
<div>
<span style="text-align: justify; text-indent: -0.25in;">
</span><br />
<div class="MsoNormal" style="text-align: justify;">
When a client connects to server,
the client gets its own thread for its connection. All the queries from that
client executed within that specified thread. The thread is cached by the
server, so they don’t need to created and destroyed for each new connection.</div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<h3 style="text-align: justify;">
</h3>
<h2>
<b>Authentication:</b></h2>
<div class="MsoNormal" style="text-align: justify;">
Whenever a client connects to a
MySQL server, the server performs the authentication in the server side. The
authentication is based on the username, host of the client and password of the
client user.</div>
</div>
<div>
<h4 style="text-align: left;">
<span style="text-align: justify; text-indent: -0.25in;"><br /></span></h4>
<h4 style="text-align: left;">
<span style="text-align: justify; text-indent: -0.25in;">Example:</span></h4>
<blockquote class="tr_bq" style="text-align: justify;">
<b>root@localhost</b><b>root : </b>Username of the client.<br />
<b>localhost :</b> host name, from
where it is originated.</blockquote>
<h2 style="text-align: justify;">
<b>Security:</b></h2>
<div class="MsoNormal" style="text-align: justify;">
After the client gets connected
successfully to MySQL server, the server will check whether that particular
client has the privileges to issue certain queries against MySQL server.</div>
<h4 style="text-align: justify;">
</h4>
<h4 style="text-align: justify;">
Example: </h4>
<blockquote class="tr_bq" style="text-align: justify;">
mysql> show privileges \G<br />
*************************** 1.
row ***************************<br />
Privilege: Alter<br />
Context: Tables<br />
Comment: To alter the table<br />
*************************** 2.
row ***************************<br />
Privilege: Alter routine<br />
Context: Functions,Procedures<br />
Comment: To alter or drop stored functions/procedures</blockquote>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<h2 style="text-align: justify;">
<b>MySQL Server layer:</b></h2>
<div class="MsoNormal" style="text-align: justify;">
This layer takes care of all the
logical functionalities of the MySQL relational database management system. The
brain of the MySQL server is resides in this layer. The logical layer of the
MySQL is divided into various sub components, which are given below:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ul>
<li>MySQL services and utilities.</li>
<li>SQL Interface.</li>
<li>SQL Parser.</li>
<li>Optimizer.</li>
<li>Caches & buffers.</li>
</ul>
<o:p></o:p><br />
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<h2 style="text-align: justify;">
<b>MySQL services and utilities:</b></h2>
<div class="MsoNormal" style="text-align: justify;">
MySQL comparatively provides wide
range of services and utilities. This is one of the main reasons for the
popularity of the MySQL. This layer provides the services and utilities for
administration and maintenance of MySQL system, some of them are mentioned
below:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ul>
<li>Backup & recovery.</li>
<li>Security.</li>
<li>Replication.</li>
<li>Cluster.</li>
<li>Partitioning.</li>
<li>Workbench.</li>
</ul>
<o:p></o:p><br />
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<h2 style="text-align: justify;">
<b>SQL Interface:</b></h2>
<div class="MsoNormal" style="text-align: justify;">
<a href="http://www.rathishkumar.in/2016/01/how-to-write-sql-query.html" target="_blank">Structured Query Language (SQL)</a>
is a query language, used to query MySQL server. It is a tool to interact
between MySQL client user and server. Some of the SQL interface components are
given below.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ul>
<li>Data Manipulation Language (DML).</li>
<li>Data Definition Language (DDL).</li>
<li>Stored Procedures.</li>
<li>Views.</li>
<li>Triggers.</li>
</ul>
<o:p></o:p><br />
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<h2 style="text-align: justify;">
<b>Parser:</b></h2>
<div class="MsoNormal" style="text-align: justify;">
MySQL parses queries to create an
internal structure (the parse tree). The
MySQL parser behaves as a single pass compiler. As per the MySQL internals, the
parser structure is given below.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ul>
<li>Lexical analysis (making words or
tokens from a character stream) is implemented at first stage, when parsing
regular statements.</li>
<li>Syntactic analysis (making
“sentences”), semantic analysis (making sure these sentences do make sense),
and code generation (for compilers) – all of them – are done at once, during
the phase of code. </li>
</ul>
<br />
<h2 style="text-align: justify;">
<b>Optimizer:</b></h2>
<div class="MsoNormal" style="text-align: justify;">
After creating the internal parse
tree, the MySQL applies a variety of optimization techniques. These techniques
may include, rewriting the query, order of scanning of tables and choosing the
right indexes to use. Actually you can ask the server to explain the various
aspects of optimization.</div>
<h4 style="text-align: justify;">
</h4>
<h4 style="text-align: justify;">
Example: </h4>
<blockquote class="tr_bq" style="text-align: justify;">
<b>EXPLAIN SELECT * FROM world.city;</b></blockquote>
<h2 style="text-align: justify;">
<b>Caches:</b></h2>
<div class="MsoNormal" style="text-align: justify;">
The MySQL cache (query cache)
stores complete result sets for <b>SELECT</b>
statements. Even before parsing the query, the MySQL server consults the query
cache. If any client issues a query that is identical to one already in the in
the cache, the server simply skip the parsing, optimization and even execution,
it just simply display the output from the cache.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<h2 style="text-align: justify;">
<b>Storage Engine Layer:</b></h2>
<div class="MsoNormal" style="text-align: justify;">
The pluggable storage engine
feature makes the MySQL as unique and preferred choice for most of the
developers. This is the feature which makes the MySQL to reach an edge over the
big player. MySQL allows us to choose the variety storage engines for different
situations and requirements. We are going to discuss about the features of each
storage engine in upcoming article, just the list of supported storage engines
are mentioned below.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ul>
<li>MyISAM.</li>
<li>InnoDB.</li>
<li>Federated.</li>
<li>Mrg_MyISAM.</li>
<li>Blackhole.</li>
<li>CSV.</li>
<li>Memory.</li>
<li>Archive.</li>
<li>Performance_schema.</li>
</ul>
<br />
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
MySQL provides this as a
pluggable storage engines, various storage engines can be used at table level.
A database can contain the tables with the multiple storage engines.<o:p></o:p></div><div class="MsoNormal" style="text-align: justify;"><br /></div>
<blockquote class="tr_bq" style="text-align: justify;">
<b>mysql>SHOW ENGINES;</b></blockquote>
<div class="MsoNormal" style="text-align: justify;">
Show engines command will list
all the storage engines supported by your server. I hope this post will give
you an overall understanding of the MySQL architecture. In next post let us
discuss about the different storage engines and their features.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
</div>
</div>
Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com34tag:blogger.com,1999:blog-2027709621250502166.post-31353810036662733932021-05-17T11:41:00.000+05:302021-05-19T11:45:42.386+05:30Face Recognition - Computing Euclidean distance in PostgreSQL<p style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px 0px 8px; text-align: left;"><span style="text-align: justify;"><span style="font-family: Roboto; font-size: medium;">In this article, we are going to discuss about implementing Euclidean distance in PostgreSQL database. Before getting into actual implementation, let me give you a quick background to understand the need for writing this article. I have been working on Face Authentication system and to perform Face Verification task, we need to compute the distance between two faces. There are a lot of implementations out there to achieve this using Python, however it did not help in my case, so I have implemented the Euclidean distance computation in PostgreSQL. Let's see about the challenges and solutions in detail.</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"></p><h3 style="text-align: justify;"><span style="font-family: Roboto; font-size: medium;">Face Recognition System:</span></h3><span class="s1" style="font-kerning: none;"><div style="text-align: justify;"><span style="font-family: Roboto; font-size: medium;">In simple terms, my implementation of Face Recognition systems consist of two parts:</span></div></span><p></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"></p><ul><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Face Registration</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Face Verification</span></span></li></ul><p></p><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Face Registration:</span></span></h3><div><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"><br /></span></span></div><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">This part contains below steps:<br /></span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"></p><ul><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Users will register their faces with name and image</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">The registered photos will be stored in folder with name as folder name</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">The user images will be feed to Convolutional Neural Network (CNN) model and extract 128 measurements for each image.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Using a K-Nearest Neighbour classifier to train a classifier with name and their corresponding 128 dimension encodings.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Save the classifier as pickle file on application directory.</span></span></li></ul><div><span style="font-family: Roboto; font-size: medium;">The CNN model we are using here is ResNet network with 29 layers and trained with about 3 million images - thanks to Davis King (dlib) for this great work and making this available to public. <a href="https://github.com/davisking/dlib-models#dlib_face_recognition_resnet_model_v1datbz2" rel="nofollow" target="_blank">Refer this page for</a> understanding more about this model.</span></div><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"><br /></span></span></h3><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Face Verification:</span></span></h3><p></p><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><br /></span></p><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;">This part contains below steps:</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"></p><ul><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">When a new user try to login with their image, it will be feed to same Convolutional Neural Network (CNN) and extract 128 dimension number vector.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">This 128 dimension vector is passed to classifier and it will be compared against all the pre-trained face encodings.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">The classifier will return name of the encodings where distance comparison value is less than 0.3<span class="Apple-converted-space"> (as I defined threshold as 0.3)</span></span></span></li></ul><div><span style="font-family: Roboto; font-size: medium;">To learn more about this Face Recognition system, please refer this <a href="https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78" rel="nofollow" target="_blank">series of insightful article</a> from Adam Geitgey. Thanks to Adam for this awesome <a href="https://github.com/ageitgey/face_recognition" rel="nofollow" target="_blank">github repository</a>, great place to start, if you are looking for open source Face Recognition system code.</span></div><div><span style="font-family: Roboto; font-size: medium;"><br /></span></div><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">The Challenges: </span></span></h3><div><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"><br /></span></span></div><span style="font-family: Roboto; font-size: medium;">This system works well, however challenges begins when we want to add new user to Face Recognition system. When a new user is registered, the encodings and name of the user should be appended to existing pre-trained data to verify user using classifier. To do this, we have to re-train the entire data - all the registered users and save the new classifier. This approach is not scalable due to following reasons:</span><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"></p><ul><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Adding new users to system is complicated as it requires training for every users.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Each time we add new user, need to train all the existing registered user images, generating 128 dimension number vector for all the images during training is time & resource consuming process, it requires higher GPU processing and doing this for each new users is not the right way.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Deleting existing user is complicated as it deals with classifier object as pickle file.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Parallelising to multiple servers with this approach is not feasible, as classifier stored as pickle file, we need to keep moving this file to all the servers, every-time we make changes to training data.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Cannot add multiple images for a person as it increases the computation.</span></span></li></ul><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">The Solution:</span></span></h3><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">After dealing with these issues for sometime, I have changed my approach to overcome above mentioned challenges. The current implementation works as follows:</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"><br /></span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"></span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqXk8PNfTHHBMk91z8EN5_9r_SSbVs56kdC2_wR4oJvK7ulGyKMW3fCJuV5kaK9F1pazd9ICvyQM1r-4cq7Fipj6mcN7QPu_r3OugiqT0HDNNCv49QyNXDRRcZSYjDx_XMnQWHt8LVUOPd/s1708/Euclidean+distance.png" style="margin-left: auto; margin-right: auto;"><img alt="Face Recognition - Computing Euclidean distance in PostgreSQL" border="0" data-original-height="1390" data-original-width="1708" height="520" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqXk8PNfTHHBMk91z8EN5_9r_SSbVs56kdC2_wR4oJvK7ulGyKMW3fCJuV5kaK9F1pazd9ICvyQM1r-4cq7Fipj6mcN7QPu_r3OugiqT0HDNNCv49QyNXDRRcZSYjDx_XMnQWHt8LVUOPd/w640-h520/Euclidean+distance.png" title="Face Recognition - Computing Euclidean distance in PostgreSQL" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Face Recognition - Computing Euclidean distance in PostgreSQL</td></tr></tbody></table><div style="text-align: center;"><span class="s1" style="font-kerning: none;"></span></div><span class="s1" style="font-kerning: none;"><span></span><span><a name='more'></a></span><span style="font-family: Roboto; font-size: medium;"><br /></span></span><p></p><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Face Registration:</span></span></h3><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"></p><ul><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">User registers with name and image</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Extract the 128 dimension number vector by feeding into Convolutional Neural Network (CNN) model with pre-trained weights.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Store name and encoding in PostgreSQL database (you can store it in any database, I will explain the reason for storing it in PostgreSQL later in the post, keep reading)</span></span></li></ul><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Face Verification:</span></span></h3><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"></p><ul><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">User login with Image</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Extract the 128 dimension number vector by feeding into Convolutional Neural Network (CNN) model with pre-trained weights.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Compute the Euclidean distance between the new user encoding with existing encodings stored in PostgreSQL using SQL statement.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Authenticate the user based on the distance value (true, if d < 03)</span></span></li></ul><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">So how this solution helps?</span></span></h3><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"></p><ul><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">We removed training part completely, as adding a new user is now INSERT operation.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">We can add any number of images per user as it's going to be inserted as new records in a table.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">We can remove/delete/update individual user records without additional overhead with DELETE & UPDATE operation.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Computations in PostgreSQL is faster as compared to Python implementation.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">We parallelise and scale this architecture to any number of users in future - currently I am performing inference with 25K encodings, that is 1:25K (this is not a big thing in database!!!)</span></span></li></ul><p></p><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Why PostgreSQL?</span></span></h3><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">By now, you will be wondering why PostgreSQL specifically, as this can be done in any SQL database. Let me give you my reasons for going with PostgreSQL.</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"></p><ul><li><span style="font-family: Roboto; font-size: medium;"><span class="s1" style="font-kerning: none;">CUBE datatype - yes I can store entire 128 dimension number vector in a single column (this is really awesome </span><span class="s2" style="font-kerning: none; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">😎</span><span class="s1" style="font-kerning: none;"> )</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">In-built function for Euclidean distance computation (<->)</span></span></li></ul><p></p><p style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;">If you are trying to do this in any other database, please mention the query in comment section, it will be helpful to others. Formula for computing euclidean distance is given below:<br /><br /></span></p><h3><span style="font-family: Roboto; font-size: medium;">Euclidean Distance:</span></h3><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoytIBe3mN5crZ2Xkbl8C4rCEoWSNCVgJOt2Lu-e8d0ximLFGYNdz-5dkMJ7KL2XPp4-Z_APMjzJr0eUht3NHWWldmWZ8ESuqDciHdS9vnH3yTd_uBanL52iS4Scw6ayHIehRXTbC41cJn/s1696/ed-formula.png" style="margin-left: auto; margin-right: auto;"><img alt="Euclidean distance for higher dimension" border="0" data-original-height="188" data-original-width="1696" height="70" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjoytIBe3mN5crZ2Xkbl8C4rCEoWSNCVgJOt2Lu-e8d0ximLFGYNdz-5dkMJ7KL2XPp4-Z_APMjzJr0eUht3NHWWldmWZ8ESuqDciHdS9vnH3yTd_uBanL52iS4Scw6ayHIehRXTbC41cJn/w640-h70/ed-formula.png" title="Euclidean distance for higher dimension" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Euclidean distance for higher dimension</td></tr></tbody></table><br /><span style="font-family: Roboto; font-size: medium;"><br />Refer this <a href="https://en.wikipedia.org/wiki/Euclidean_distance" rel="nofollow" target="_blank">wikipedia</a> page for more details. Okay, let's jump into implementation.</span><p></p><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><h3 style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Installing PostgreSQL from source:</span></span></h3><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"></p><div style="text-align: justify;"><span style="font-family: Roboto; font-size: medium;">If you have installed PostgreSQL binary, you will not be able to achieve this as the default max dimension of CUBE datatype is 100 in binary installation. To change the CUBE max dimension to 128, we need to modify the source code file and install it.</span></div><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"><div style="text-align: justify;"><br /></div><div style="text-align: justify;">Don’t worry, if you have not done this before, it is a simple task, follow the below steps correctly, you will be done in no time.</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">I will show you on how to do it on Linux/MacOS machine, if you are doing it on Windows, please mention the steps in comment section, it will be helpful to other readers.</div></span></span><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"></p><ul><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Download the latest PostgreSQL source code files from here: <a href="https://www.postgresql.org/ftp/source/"><span class="s3" style="font-kerning: none; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">https://www.postgresql.org/ftp/source/</span></a><span class="Apple-converted-space"> </span></span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">The version, I am installing is <a href="https://ftp.postgresql.org/pub/source/v13.2/postgresql-13.2.tar.bz2"><span class="s3" style="font-kerning: none; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">postgresql-13.2.tar.bz2</span></a> (March, 2021) - you can install the latest version.</span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Unzip the source code to local directory: tar -xvjf <a href="https://ftp.postgresql.org/pub/source/v13.2/postgresql-13.2.tar.bz2"><span class="s3" style="font-kerning: none; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;">postgresql-13.2.tar.bz2</span></a></span></span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Edit the cubedata.h file located at /contrib/cube/cubedata.h: </span></span>vi <span style="color: #56b6c2; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">/</span><span style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">contrib</span><span style="color: #56b6c2; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">/</span><span style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">cube</span><span style="color: #56b6c2; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">/</span><span style="background-color: #282c34; color: #abb2bf; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 13px; white-space: pre;">cubedata.h</span></li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Change the CUBE_MAX_DIM </span></span><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">to value to 128 from 100, after changing, it looks like: </span></span>#define CUBE_MAX_DIM (128)</li><li><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Run the below command in order - this is short version of installation from source code, if you are looking for detailed steps, visit official documentation: <span class="s3" style="font-kerning: none; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"><a href="https://www.postgresql.org/docs/12/install-requirements.html">https://www.postgresql.org/docs/12/install-requirements.html</a></span></span></span></li></ul><div><pre class="lang-java s-code-block hljs" style="border-radius: 5px; border: 0px; box-sizing: inherit; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: 0px; margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><span face="Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, sans-serif">./configure
make
su
make install
adduser postgres
mkdir /usr/local/pgsql/data
chown postgres /usr/local/pgsql/data
su - postgres
/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data
/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l
\logfile start
/usr/local/pgsql/bin/createdb test
/usr/local/pgsql/bin/psql test</span></pre></div><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"></p><div style="text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><br /></span></div><span class="s1" style="font-kerning: none;"><div style="text-align: justify;"><span style="font-family: Roboto; font-size: medium;">Now you are done with PostgreSQL installations, we need to add CUBE extension to get this CUBE datatype feature.<span class="Apple-converted-space"> </span></span></div></span><p></p><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: left;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Go to the </span><span><span style="color: #abb2bf; font-family: Menlo, Monaco, Courier New, monospace;"><span style="background-color: #282c34; font-size: 13px; white-space: pre;">contrib/cube</span></span><span style="font-family: Roboto; font-size: medium;"> directory and install the extension by running the below commands:</span></span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: left;"><span class="s1" style="font-kerning: none;"><span><span style="font-family: Roboto; font-size: medium;"><br /></span></span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Roboto; font-size: medium;">make</span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span style="font-family: Roboto; font-size: medium;">make install</span></p><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><br /></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Lets create a database and user for our Face Recognition system:</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"><br /></span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"></span></span></p><pre class="lang-java s-code-block hljs" style="border-radius: 5px; border: 0px; box-sizing: inherit; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: 0px; margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><span face="Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, sans-serif">su - postgres
/usr/local/pgsql/bin/createdb facedb
/usr/local/pgsql/bin/psql facedb
CREATE USER face WITH PASSWORD = ‘face’;
GRANT ALL PRIVILEGES ON DATABASE facedb TO face;</span></pre><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"></span></span></p><div style="text-align: justify;"><br /></div><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">To use CUBE extension from facedb we need to create the extension under this database, to do that, run the following SQL statement:</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"><br /></span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"></span></span></p><pre class="lang-java s-code-block hljs" style="border-radius: 5px; border: 0px; box-sizing: inherit; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: 0px; margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><span face="Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, sans-serif">CREATE EXTENSION cube;</span></pre><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Create a table for storing Employee ID, Name, Encoding - you can skip the employeeid column - I am adding it to get Employee ID in addition with Name at the Face Verification:</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"><br /></span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"></span></span></p><pre class="lang-java s-code-block hljs" style="border-radius: 5px; border: 0px; box-sizing: inherit; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: 0px; margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><span face="Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, sans-serif">CREATE TABLE encodings
(
encodeId serial,
employeeid char(8) default '00000000',
employeename varchar,
encodevector cube
);</span></pre><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Also create an index on CUBE column to retrieve the data faster, as follows:</span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"><br /></span></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;"></span></span></p><pre class="lang-java s-code-block hljs" style="border-radius: 5px; border: 0px; box-sizing: inherit; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: 0px; margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><span face="Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, sans-serif">CREATE INDEX encodevector_idx ON encodings(encodevector);</span></pre><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p3" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; text-align: justify;"><span class="s1" style="font-kerning: none;"><span style="font-family: Roboto; font-size: medium;">Great!!!, we are done with database part, lets create Python function to call from registration and verification code:</span></span></p><p class="p4" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 13px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><span class="s1" style="font-kerning: none;"></span><br /></span></p><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;">Import the required packages:</span></p><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><br /></span></p><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"></span></p><pre class="lang-java s-code-block hljs" style="border-radius: 5px; border: 0px; box-sizing: inherit; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: 0px; margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><span face="Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, sans-serif">from psycopg2.pool import SimpleConnectionPool
from contextlib import contextmanager</span></pre><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: large;">Define the database and connection parameters:</span></p><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><br /></span></p><pre class="lang-java s-code-block hljs" style="border-radius: 5px; border: 0px; box-sizing: inherit; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: 0px; margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><span face="Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, sans-serif">hostname = 'localhost'
dbname = 'facedb'
dbuser = 'face'
dbpass = 'face'
db = SimpleConnectionPool(1, 10,host=hostname,database=dbname,user=dbuser,
password=dbpass)
@contextmanager
def get_connection():
con = db.getconn()
try:
yield con
finally:
db.putconn(con)</span></pre><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: large;"><br /></span></p><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: large;">Define the function to INSERT encodings, name to PostgreSQL table:</span></p><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><br /></span></p><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><br /></span></p><pre class="lang-java s-code-block hljs" style="border-radius: 5px; border: 0px; box-sizing: inherit; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: 0px; margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><span face="Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, sans-serif">def saveDb(employeeId, name, face_encodings):
with get_connection() as conn:
try:
cursor = conn.cursor()
query = "INSERT INTO encodings(employeeid, employeename, encodevector) VALUES ('{}','{}', CUBE(array[{}]))".format(employeeId, name, ','.join(str(s) for s in face_encodings))
cursor.execute(query)
cursor.close()
conn.commit()
return 1
except:
conn.rollback()
return 0</span></pre><pre class="lang-java s-code-block hljs" style="border-radius: 5px; border: 0px; box-sizing: inherit; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: 0px; margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><span face="Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, sans-serif"><br /></span></pre><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;">Function to compare the encoding with existing encodings from table and returns the corresponding name and employee ID</span></p><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><br /></span></p><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"></span></p><pre class="lang-java s-code-block hljs" style="border-radius: 5px; border: 0px; box-sizing: inherit; font-stretch: inherit; font-variant-east-asian: inherit; font-variant-numeric: inherit; line-height: 1.30769; margin-bottom: 0px; margin-top: 0px; max-height: 600px; overflow-wrap: normal; overflow: auto; padding: 12px; vertical-align: baseline; width: auto;"><span face="Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, sans-serif">def findDb(face_encodings, threshold=0.4):
with get_connection() as conn:
try:
name = 'Unknown'
empid = '0000000'
cursor = conn.cursor()
query = "SELECT employeename, employeeid FROM encodings WHERE sqrt(power(CUBE(array[{}]) <-> encodevector, 2)) <= {} ".format(','.join(str(s) for s in face_encodings), threshold) + "ORDER BY sqrt(power(CUBE(array[{}]) <-> encodevector, 2)) ASC LIMIT 1".format(','.join(str(s) for s in face_encodings))
cursor.execute(query)
data = cursor.fetchone()
if data == None:
return name, empid
else:
return data
except Exception as e:
return e</span></pre><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;"><br /></span></p><p class="p8" style="font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; min-height: 15px; text-align: justify;"><span style="font-family: Roboto; font-size: medium;">I hope this article is helpful to you. Please write in the comment section, if you are facing any challenges with this implementation. I will be glad to assist you.</span></p><p></p>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com8tag:blogger.com,1999:blog-2027709621250502166.post-77679293160699111142019-06-10T17:21:00.002+05:302021-03-06T13:48:48.965+05:30SQL Server Always On Availability Group (AG) Listener: step by step guide<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Availability databases hosted on SQL
Server Always On Availability Group (AG) can be connected using a unique
Virtual Network Name (VNN) called Availability Group Listener. When
Availability Group is enabled, clients can connect to databases in both primary
and secondary replicas without explicitly specifying the SQL Server instance
name. You don’t even need to know the instance name to connect to Availability
Group (AG).<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">When you have configured read-only
routing for secondary replicas and the application or client connection
contains the application intent as read-only, the listener will redirect the
connection to secondary readable replicas, Otherwise, the listener redirects
both read-write and read-only intent connections to primary replica. If there
is a fail-over, the listener will redirect connections to new primary and
secondary connections based on the application intent.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span>
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><i>Related articles in Always On Availability Group:</i></span><br />
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><i><a href="https://www.rathishkumar.in/2019/05/How-to-configure-Read-Only-routing-on-SQL-Server-2016-Always-On-Availability-Group-AG.html" target="_blank">Always On Availability Group Read-Only Routing</a></i></span><br />
<i><a href="https://www.rathishkumar.in/2019/05/understanding-and-configuring-quorum.html" target="_blank">Windows Server Failover Cluster Quorum configuration for SQL Server Always On</a></i><br />
<i><a href="https://www.rathishkumar.in/2019/06/step-by-step-guide-to-add-ssidb-to-alwayson-ag.html" target="_blank">Adding SSISDB to Always On Availability Group</a></i><br />
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Availability group listener consist of
following objects:<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ul>
<li><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Domain Name Systems (DNS)</span></li>
<li><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Listener Port</span></li>
<li><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">One or more IP addresses (VIP)</span></li>
</ul>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><span></span>The Listener is always owned by the
SQL Server instance where the primary replica resides. At the time of failover,
the new primary replica will own the listener.</span><br />
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
Example:
<b><o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ul>
<li><b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">DNS:
</span></b><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">salesag.mscorp.com<o:p></o:p></span></li>
<li><b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Port:
</span></b><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">1433<o:p></o:p></span></li>
<li><b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">IP:
</span></b><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">static
or DHCP</span></li>
</ul>
<h3 style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Creating Availability Group Listener:</span></b></h3>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Availability Group Listener can be
created while creating Availability Group or it can be created post creating
the AG. The Listener creation options are available on <b style="mso-bidi-font-weight: normal;">Specify Replicas</b> window and under the <b style="mso-bidi-font-weight: normal;">Listener tab</b>.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">If you want to skip the Listener
creation while creating Availability Group, you can leave the default option <b style="mso-bidi-font-weight: normal;">Do not create an availability group
listener now</b> check the below image for reference:<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQMjezuc4d8ULOKevcwD26tIkM2AnZzFTlxUR9Z4O_borpmGBBXuQ-XGGi4dn2V5H1n9f5aZv0yYf8C6zAzCY12FhcJLcFxVNTcQU0u81F6KyriTjQoSmYLj95otK9DpjzCyu3G3Cz_yEt/s1600/1.PNG" style="margin-left: auto; margin-right: auto;"><img alt="New Availability Group - Specify Replicas" border="0" data-original-height="720" data-original-width="825" height="558" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQMjezuc4d8ULOKevcwD26tIkM2AnZzFTlxUR9Z4O_borpmGBBXuQ-XGGi4dn2V5H1n9f5aZv0yYf8C6zAzCY12FhcJLcFxVNTcQU0u81F6KyriTjQoSmYLj95otK9DpjzCyu3G3Cz_yEt/s640/1.PNG" title="New Availability Group - Specify Replicas" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>New Availability Group - Specify Replicas</b></td></tr>
</tbody></table>
<span style="text-align: justify;"><br /></span>
<br />
<div>
<span style="text-align: justify;"><span><a name='more'></a></span>Post creating the Availability Group,
you can configure listener on </span><b style="text-align: justify;">Availability
Groups -> Group Name -> Add Listener.</b><br />
<div class="MsoNormal" style="text-align: justify;">
<br />
Either on <b>create an availability group listener</b> or <b>New Availability Group Listener</b> wizard, specify the <b>Listener DNS Name, Port, Network Mode</b> values.
The following figure shows the example availability group listener from my lab.</div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgw-xTwzkMGHetOp3QOmEMuNMLVckd8Qc1QvPpNhzn5fu_S_ujIRCBAY90niARY1sF3mYRgthGg0zyOaS4MNeI-nq1muWtDjZgd0IGAwqoD0sPQTQsI73Xii27ANSQwrpAt4iYDuup8Vezn/s1600/2.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Create availability group listener" border="0" data-original-height="392" data-original-width="514" height="488" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgw-xTwzkMGHetOp3QOmEMuNMLVckd8Qc1QvPpNhzn5fu_S_ujIRCBAY90niARY1sF3mYRgthGg0zyOaS4MNeI-nq1muWtDjZgd0IGAwqoD0sPQTQsI73Xii27ANSQwrpAt4iYDuup8Vezn/s640/2.PNG" title="Create availability group listener" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Create availability group listener</b></td></tr>
</tbody></table>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">You can also, configure Network Mode
as DHCP (Dynamic Host Configuration Protocol). DHCP is limited to single subnet
and it is not recommended for production environments. Choose the Static IP
option from the drop down, if you want the multi-subnet availability group. <o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">To Add IP address, click on <b style="mso-bidi-font-weight: normal;">Add</b> button and provide the IP address
at <b style="mso-bidi-font-weight: normal;">IPv4 Address</b> text box.<o:p></o:p></span></div>
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaD9vH_NEv6l27lXvD1ysUrIIN6uDjwyXQg9F3SAMqPCfIoCTdW-osQHhvzpGNe7h5nLBc5pTMg0uA6sIphK4l82uN1-TmpLyv2rh2B2lP9iHxMNf0P3oM_YRYR-1oVE90RIbYvGVbZDTa/s1600/3.PNG" style="margin-left: auto; margin-right: auto;"><img alt="You can also, configure Network Mode as DHCP (Dynamic Host Configuration Protocol). DHCP is limited to single subnet and it is not recommended for production environments. Choose the Static IP option from the drop down, if you want the multi-subnet availability group. To Add IP address, click on Add button and provide the IP address at IPv4 Address text box." border="0" data-original-height="316" data-original-width="538" height="374" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaD9vH_NEv6l27lXvD1ysUrIIN6uDjwyXQg9F3SAMqPCfIoCTdW-osQHhvzpGNe7h5nLBc5pTMg0uA6sIphK4l82uN1-TmpLyv2rh2B2lP9iHxMNf0P3oM_YRYR-1oVE90RIbYvGVbZDTa/s640/3.PNG" title="You can also, configure Network Mode as DHCP (Dynamic Host Configuration Protocol). DHCP is limited to single subnet and it is not recommended for production environments. Choose the Static IP option from the drop down, if you want the multi-subnet availability group. To Add IP address, click on Add button and provide the IP address at IPv4 Address text box." width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Create availability group listener - Add IP Address wizard</b></td></tr>
</tbody></table>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Additionally, you can script out the
changes to query windows and see T-SQL script. Codes from my demo cluster:<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<blockquote class="tr_bq" style="line-height: normal; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">USE</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;"> [master]<br /><o:p></o:p></span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">GO</span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">ALTER</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;"> </span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">AVAILABILITY</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;"> </span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">GROUP</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;">
[MSCORPAG]<br /><o:p></o:p></span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">ADD</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;"> </span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">LISTENER</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;"> </span><span style="color: red; font-family: "consolas"; font-size: 9.5pt;">N'mscorpag'</span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt;"> </span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt;">(</span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">WITH</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;"> </span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">IP</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt;">((</span><span style="color: red; font-family: "consolas"; font-size: 9.5pt;">N'192.168.0.7'</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt;">,</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;"> </span><span style="color: red; font-family: "consolas"; font-size: 9.5pt;">N'255.255.252.0'</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt;">)</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt;">)</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt;">,</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;"> </span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">PORT</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt;">=</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;">1433</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt;">);</span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">GO</span></blockquote>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">You can configure port 1433, default
SQL Server instance port for availability group listener. In that case, you
don’t have to specify the port number explicitly on connection string or client
connection. If you specify custom port, you have to explicitly specify it. If
you have more than one SQL Server instances on the machine, I recommend you to
use different port, because, the listener may be configured to different
instance which is listening on different port.</span><br />
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<h3 style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Connecting Availability Group
Listener:</b></span></h3>
<div class="MsoNormal" style="text-align: justify;">
To connect availability databases
through availability group listener, specify the listener name on server name
box as below:</div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigMcIsPAEQS6-WQb_iF6MJas2YLcUICmgFK4GtynKi02IzmKLxUhoWrCEUUuKeSsT4j-eZfjT-gHkluVcxuOBk5cTrU50DR0p3dGvqo3xszYBy0WbYKPOUvsOxb5x1Zsy2i_X6I3_65nvr/s1600/4.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Connecting to availability group listener in SSMS" border="0" data-original-height="319" data-original-width="481" height="424" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigMcIsPAEQS6-WQb_iF6MJas2YLcUICmgFK4GtynKi02IzmKLxUhoWrCEUUuKeSsT4j-eZfjT-gHkluVcxuOBk5cTrU50DR0p3dGvqo3xszYBy0WbYKPOUvsOxb5x1Zsy2i_X6I3_65nvr/s640/4.PNG" title="Connecting to availability group listener in SSMS" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Connecting to availability group listener in SSMS</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal">
I am using default port number, if you
are using different port number, specify explicitly on the connection string.
Example connection string with different port number provided below:<o:p></o:p></div>
</div>
<br />
<blockquote class="tr_bq">
<span style="background: rgb(250, 250, 250); font-family: "courier new"; font-size: 10.5pt; line-height: 107%;">connUrl = </span><span style="background: rgb(250, 250, 250); color: #a31515; font-family: "courier new"; font-size: 10.5pt; line-height: 107%;">"jdbc:sqlserver://mscorpag.mscorp.com:3306;databaseName=msdb;user=rathish;password=pwd"</span><span style="background: rgb(250, 250, 250); font-family: "courier new"; font-size: 10.5pt; line-height: 107%;">;</span></blockquote>
<h3 style="text-align: justify;">
<b>Listener and user permissions:</b></h3>
<div class="MsoNormal" style="text-align: justify;">
When you create an availability group
listener, the cluster will create a computer object on domain controller and
assign its computer name as virtual network name automatically, to perform
this, the cluster should have <b>Create
Computer Object</b> permission on your Active Directory. <o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Additionally, I have created
availability group listener with the help of Domain Administrator by following
below method.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Create a computer object on Active
Directory and assign listener name as computer name to that object. The cluster
(name of the cluster which own the availability group) will have the full
control on the newly created computer object. Add an entry at <b>Domain Name
Systems (DNS)</b> against that computer name with a static IP address. Now create
listener on availability group. In troubleshooting section of this article, we
look into the common error scenarios with listener creation permission.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
On SQL Server, you need <b>sysadmin</b> privilege to create the availability
group listener and at least <b>ALTER
AVAILABILITY GROUP</b> permission to modify the listener.<br />
<br /></div>
<h3 style="text-align: justify;">
<b>Read-Only Routing and availability
group listener:</b></h3>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
In SQL Server Always On Availability
Group (AG), read-only routing features provides the scalability by redirecting
read-only connections (SELECT queries) to readable secondary replicas. Routing
of connection to secondary replicas works, only when the applications or
clients connecting availability databases through availability group listener. If
you are connecting directly to SQL Server instance name, at the time of
failover, connections will not automatically failover to new primary server.</div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
As stated earlier, listener will redirect
connections to both primary and secondary replicas. If you have configured any
of the secondary replicas as readable and specify the application intent as
read only, the listener will redirect connections secondary readable replicas
based on the read-only routing configuration. Refer my previous article <a href="https://www.rathishkumar.in/2019/05/How-to-configure-Read-Only-routing-on-SQL-Server-2016-Always-On-Availability-Group-AG.html" target="_blank">How toconfigure Read-Only routing on SQL Server Always On Availability Group</a> for
detailed explanation. Example connection string with Application Intent and
default database given below:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<blockquote class="tr_bq">
<span style="background: rgb(250, 250, 250); font-family: "courier new"; font-size: 10.5pt; line-height: 107%;">connUrl = </span><span style="background: rgb(250, 250, 250); color: #a31515; font-family: "courier new"; font-size: 10.5pt; line-height: 107%;">"jdbc:sqlserver://mscorpag.mscorp.com:3306;databaseName=mscorp;user=rathish;password=pwd;ApplicationIntent=ReadOnly"</span><span style="background: rgb(250, 250, 250); font-family: "courier new"; font-size: 10.5pt; line-height: 107%;">;</span></blockquote>
<div class="MsoNormal" style="text-align: justify;">
You must specify the ApplicationIntent
and Default database on connection string for read-only routing to work,
otherwise the connection will be redirected to primary replica only, later in
this article, we will see the troubleshooting steps for this scenario.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<h3 style="text-align: justify;">
<b>Availability Group Listener and
multi-subnet failover:</b></h3>
<div class="MsoNormal" style="text-align: justify;">
You should set the MultiSubnetFailover
option as True, enabling this option provides faster failover when your
availability group spans over multiple subnets. It is recommended to set this
option as True, even, if the availability group only spans a single subnet.
This provides additional optimisation, even for single subnets ate the time of
failover.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Example connection string with
multi-subnet failover option:<o:p></o:p></div>
<div class="MsoNormal">
<span style="background: rgb(250, 250, 250); font-family: "courier new"; font-size: 10.5pt; line-height: 107%;"><br /></span></div>
<blockquote class="tr_bq">
<span style="background: rgb(250, 250, 250); font-family: "courier new"; font-size: 10.5pt; line-height: 107%;">connUrl = </span><span style="background: rgb(250, 250, 250); color: #a31515; font-family: "courier new"; font-size: 10.5pt; line-height: 107%;">"jdbc:sqlserver://mscorpag.mscorp.com:3306;databaseName=mscorp;user=rathish;password=pwd;ApplicationIntent=ReadOnly;MultiSubnetFailover=True"</span><span style="background: rgb(250, 250, 250); font-family: "courier new"; font-size: 10.5pt; line-height: 107%;">;</span> </blockquote>
<h3 style="text-align: justify;">
<b>Monitoring Availability Group
Listener:</b></h3>
<div class="MsoNormal" style="text-align: justify;">
SQL Server provides following catalog
views to monitor availability group listener properties:<br />
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<blockquote class="tr_bq" style="text-align: justify;">
<span style="color: lime; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">sys</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">.</span><span style="color: lime; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">availability_group_listener_ip_addresses:</span> returns the
virtual IP address of availability group listeners<br />
<span style="color: lime; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">sys</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">.</span><span style="color: lime; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">availability_group_listeners:</span> returns
the network name of availability group listeners<br />
<span style="color: lime; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">sys</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">.</span><span style="color: lime; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">dm_tcp_listener_states:</span> returns
the status of TCP IP address and port of availability group listener.<span style="text-align: left;"> </span></blockquote>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<h3 style="text-align: justify;">
<b>Removing Availability Group Listener:</b></h3>
<div class="MsoNormal" style="text-align: justify;">
You can remove availability group
listener, either using SSMS or using T-SQL. In SSMS, right-click on <b>listener
name</b> and select the <b>Delete </b>options to remove the availability group listener.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Using <b>T-SQL:</b><o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<div class="MsoNormal" style="text-align: justify;">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;"><br /></span></div>
<blockquote class="tr_bq" style="text-align: justify;">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">ALTER</span><span style="font-family: "consolas"; font-size: 9.5pt; line-height: 107%;"> </span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">AVAILABILITY</span><span style="font-family: "consolas"; font-size: 9.5pt; line-height: 107%;"> </span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">GROUP</span><span style="font-family: "consolas"; font-size: 9.5pt; line-height: 107%;"> MSCORPAG </span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">REMOVE</span><span style="font-family: "consolas"; font-size: 9.5pt; line-height: 107%;"> </span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">LISTENER</span><span style="font-family: "consolas"; font-size: 9.5pt; line-height: 107%;"> </span><span style="color: red; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">'mscorpag.mscorp.net'</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">;</span></blockquote>
<br />
<div style="text-align: justify;">
<h3 style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Troubleshooting
availability group listener issues:</span></b></h3>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Scenario
1: Unable to create availability group listener:<o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Error:</b> Microsoft SQL Server, Error:
19457<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Message:
<o:p></o:p></span></b></div>
<div class="MsoNormal" style="background: white; mso-background-themecolor: background1; text-align: justify;">
<span style="background: rgb(242, 242, 242); color: black;"><br /></span></div>
<blockquote class="tr_bq" style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; text-align: justify;">
<span style="background-color: white;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: black;"><i>The specified IP Address '<ip address="">' is not valid in the cluster-allowed IP range. Check with the
network administrator to select values that are appropriate for the
cluster-allowed IP range. (Microsoft SQL Server, Error: 19457)</ip></i></span></span></blockquote>
<blockquote class="tr_bq">
<i><span style="background-color: white;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: black;">Msg 19471, Level 16, State 0, Line 2</span></span><span style="background-color: white;"><span style="color: black; mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial;"><span style="-webkit-text-stroke-width: 0px; float: none; font-variant-caps: normal; font-variant-ligatures: normal; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; widows: 2; word-spacing: 0px;">The WSFC cluster could not bring the Network Name resource
with DNS name '<dns name="">' online. The DNS name may have been taken or
have a conflict with existing name services, or the WSFC cluster service may
not be running or may be inaccessible. Use a different DNS name to resolve name
conflicts, or check the WSFC cluster log for more information.</dns></span></span></span></span></i></blockquote>
<blockquote class="tr_bq">
<i><span style="background-color: white;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: black;">Msg 19476, Level 16, State 4, Line 2</span></span><span style="background-color: white;"><span style="color: black; mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial;"><span style="-webkit-text-stroke-width: 0px; float: none; font-variant-caps: normal; font-variant-ligatures: normal; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; widows: 2; word-spacing: 0px;">The attempt to create the network name and IP address for the
listener failed. The WSFC service may not be running or may be inaccessible in
its current state, or the values provided for the network name and IP address
may be incorrect. Check the state of the WSFC cluster and validate the network
name and IP address with the network administrator.</span></span></span></span></i></blockquote>
<div class="MsoNormal" style="background: white; mso-background-themecolor: background1; text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Possible Causes:<o:p></o:p></span></b></div>
<div class="MsoNormal" style="background: white; mso-background-themecolor: background1; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="background: white; mso-background-themecolor: background1; text-align: justify;">
<b>Cause 1:</b> The cluster name account does not have the Create
Computer Objects in Active Directory Organizational Unit.<o:p></o:p></div>
<div class="MsoNormal" style="background: white; mso-background-themecolor: background1; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="background: white; mso-background-themecolor: background1; text-align: justify;">
<b>Solution:</b> Grant <b>Create Computer Objec</b>t to Cluster account name.<o:p></o:p></div>
<div class="MsoNormal" style="background: white; mso-background-themecolor: background1; text-align: justify;">
<h3>
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><o:p> </o:p></span><br /><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>How to grant Create Computer Objects to Cluster name account:</b></span></h3>
</div>
<div class="MsoNormal" style="background: white; mso-background-themecolor: background1; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Select the <b style="mso-bidi-font-weight: normal;">Advanced Feature</b>
option by <b style="mso-bidi-font-weight: normal;">View -> Advanced Features:</b><o:p></o:p></span></div>
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_439YEY4mp4ufQyE3QHSJUo0iOw5OorfmkHD-FNAm4md3wp_z3313d2Zmyt54QjkGiuQySa0wv2dhhXWePIG0EkJfB63XzXIl6LtYXFx7IbUiPHQ7D8QXUSaQAPEBOxJ4Ek4cYbJAgSWI/s1600/5.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Troubleshooting Availability Group Listener - View Advanced Feature on AD" border="0" data-original-height="530" data-original-width="753" height="450" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_439YEY4mp4ufQyE3QHSJUo0iOw5OorfmkHD-FNAm4md3wp_z3313d2Zmyt54QjkGiuQySa0wv2dhhXWePIG0EkJfB63XzXIl6LtYXFx7IbUiPHQ7D8QXUSaQAPEBOxJ4Ek4cYbJAgSWI/s640/5.PNG" title="Troubleshooting Availability Group Listener - View Advanced Feature on AD" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Troubleshooting Availability Group Listener - View Advanced Feature on AD</b></td></tr>
</tbody></table>
<div style="text-align: justify;">
<div class="MsoNormal" style="background: white;">
<br />
Go to <b>Advanced Security Settings</b> for <b>Organisational Unit (OU)</b>:
Follow the highlighted part as below image:<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2worS2kYXFeeYDj4V5HhKLpm9U-BttFOxHsArxyfkLfM9BpgEnoQyInErmxxUQc5le_PaAPdhBRS2G4n3qKXKcXXSCTCAlFqy2BT8cJpTsyey8Eo4AZHnUBNxRC4Y3A-_biqj9-Vx-Cd5/s1600/6.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Advanced Security Settings of Organizational Unit" border="0" data-original-height="623" data-original-width="1366" height="290" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2worS2kYXFeeYDj4V5HhKLpm9U-BttFOxHsArxyfkLfM9BpgEnoQyInErmxxUQc5le_PaAPdhBRS2G4n3qKXKcXXSCTCAlFqy2BT8cJpTsyey8Eo4AZHnUBNxRC4Y3A-_biqj9-Vx-Cd5/s640/6.PNG" title="Advanced Security Settings of Organizational Unit" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Advanced Security Settings of Organizational Unit</b></td></tr>
</tbody></table>
<div style="text-align: justify;">
<div class="MsoNormal" style="background: white;">
<br />
Select the <b>Create Computer Objects</b> from permission tab:</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizYWeO47iDi79H_WD-J-XL-TPZLjRgzHTE01N7B8235123bIXsV8QTdRTBmyRFnCzHxo052RrEmVzzGtl3gbUDj5S_RQnr19n53vMIZ_Ehat732FPaZ5oAQDNP_mT7SDYyv60TAJUX8Dpw/s1600/7.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Create Computer Objects permission to cluster name account" border="0" data-original-height="593" data-original-width="524" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizYWeO47iDi79H_WD-J-XL-TPZLjRgzHTE01N7B8235123bIXsV8QTdRTBmyRFnCzHxo052RrEmVzzGtl3gbUDj5S_RQnr19n53vMIZ_Ehat732FPaZ5oAQDNP_mT7SDYyv60TAJUX8Dpw/s640/7.PNG" title="Create Computer Objects permission to cluster name account" width="564" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Create Computer Objects permission to cluster name account</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Cause 2:</b> The cluster user account, which has Account Operator
permission on Active Directory by default can create up to 10 computer objects,
and it is exceeds its limit now.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Solution:</b> If your organisation polity allowed, grant Create
Computer Object permission to cluster account, or prestage the required objects
on the Active Directory, Organizational Unit (OU).<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<h3>
<b>How to prestage computer objects for availability group listener:</b></h3>
</div>
<div class="MsoNormal" style="background: white;">
When you create an availability group listener, a virtual computer
object will be created at the particular organizational unit of active
directory automatically.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
You can <b>prestage</b> the virtual computer objects as follows:<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<span face=""calibri" , sans-serif" style="font-size: 11pt; line-height: 107%;"><br /></span></div>
<div class="MsoNormal" style="background: white;">
<span face=""calibri" , sans-serif" style="font-size: 11pt; line-height: 107%;">Create
a computer objects under the organizational unit where you hosted the cluster,
check with your domain administrator, if you don’t have access to create
objects on <b>Active Directory</b> Domain Controller</span></div>
<div class="MsoNormal" style="background: white;">
<span face=""calibri" , sans-serif" style="font-size: 11pt; line-height: 107%;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB9xl0p8hgwoval1q0QfUYt9KHz7F_dCIbEzUIWPgAXg_5WpQ8doKGUYD1FsDH-mumAqwf-5WZzLZvf-NIYNuveQvRctNTvag3vsBN3sp_jnDWVaoyECm1wLNYileoL_swT3baAT68xY4t/s1600/8.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Add virtual computer objects on Active Directory for Availability Group listener" border="0" data-original-height="536" data-original-width="752" height="456" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB9xl0p8hgwoval1q0QfUYt9KHz7F_dCIbEzUIWPgAXg_5WpQ8doKGUYD1FsDH-mumAqwf-5WZzLZvf-NIYNuveQvRctNTvag3vsBN3sp_jnDWVaoyECm1wLNYileoL_swT3baAT68xY4t/s640/8.PNG" title="Add virtual computer objects on Active Directory for Availability Group listener" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Add virtual computer objects on Active Directory for Availability Group listener</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="background: white;">
<span face=""calibri" , sans-serif" style="font-size: 11pt; line-height: 107%;"><br /></span></div>
<div class="MsoNormal" style="background: white;">
<span face=""calibri" , sans-serif" style="font-size: 11pt; line-height: 107%;"></span></div>
<div class="MsoNormal" style="background: white;">
Grant <b>Full control</b> permission to cluster name account on newly
created computer object. Note that, this name should be same as the listener
name, you will be creating on availability group.<o:p></o:p></div>
</div>
<div style="text-align: justify;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjP8XikP2pSg0eEb7eNNBMUfEYicHLtnGyGZpDgvDuRvPyfMB3TcNZ32Z0r3iiBUucF72ShKCauTCFT27F-ipWw2Ns7oIi971_w0y3tkvNqBc7FEXQmjqpj8ZFeF68oRkr9M44HPz4HIvpw/s1600/9.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Full control permission on virtual computer object to cluster name account" border="0" data-original-height="573" data-original-width="796" height="460" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjP8XikP2pSg0eEb7eNNBMUfEYicHLtnGyGZpDgvDuRvPyfMB3TcNZ32Z0r3iiBUucF72ShKCauTCFT27F-ipWw2Ns7oIi971_w0y3tkvNqBc7FEXQmjqpj8ZFeF68oRkr9M44HPz4HIvpw/s640/9.PNG" title="Full control permission on virtual computer object to cluster name account" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Full control permission on virtual computer object to cluster name account</b></td></tr>
</tbody></table>
<div style="text-align: justify;">
<div class="MsoNormal" style="background: white;">
<b><br /></b>
<b>Scenario 2: Read-Only
routing not working when connecting to listener</b><o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Error: </b><i>ApplicationIntent=ReadOnly</i> specified on application
connection string and read-only routing configured on the availability group,
but when connecting to listener, it is not redirecting to secondary readable
replicas.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Possible Cause:</b> You have not mentioned the default database on the
connection string. For more information on this behaviour can be found <a href="https://www.rathishkumar.in/2019/05/How-to-configure-Read-Only-routing-on-SQL-Server-2016-Always-On-Availability-Group-AG.html" target="_blank">here</a>.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Solution:</b> Add <i>Initial Catalog </i>value in connection string.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
Example <b>SSMS </b>connection string:</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH0DmRYnxdTafBT9qrEgFMqznASARUqgN4ADohr-5EdhfnpeD0fVFd2_iEKcucU4UQpBp848vsuesIilVfwbg4PBNnvJnFhL0x-2FgG0xOP1OjyMXxOueGQG_PCw0sRD0v5ZwQ7zXpxzU5/s1600/10.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Read-Only Routing Connection parameters on SSMS" border="0" data-original-height="518" data-original-width="477" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjH0DmRYnxdTafBT9qrEgFMqznASARUqgN4ADohr-5EdhfnpeD0fVFd2_iEKcucU4UQpBp848vsuesIilVfwbg4PBNnvJnFhL0x-2FgG0xOP1OjyMXxOueGQG_PCw0sRD0v5ZwQ7zXpxzU5/s640/10.PNG" title="Read-Only Routing Connection parameters on SSMS" width="588" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Read-Only Routing Connection parameters on SSMS</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="background: white;">
<b><br /></b>
<b>Scenario 3: Availability Group
Listener - login time-out error<o:p></o:p></b></div>
<div class="MsoNormal" style="background: white;">
<b><br /></b></div>
<div class="MsoNormal" style="background: white;">
<b>Error:</b> You are unable to connect to availability group listener in
a multi-subnet environment. This error usually occurs at the time of failover.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Possible Causes:</b><o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<b><br /></b></div>
<div class="MsoNormal" style="background: white;">
<b>Cause 1:</b> Your application uses legacy data provider that does not
support the multi-subnet failover features.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Solution: </b>Use the newer version of <b>SQLClient</b> drivers that supports
multi-subnet features.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Cause 2:</b> <i>MultiSubnetFailover</i> parameters are not used on the
connection string.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Solution: </b>Include <i>MultiSubnetFailover=True</i> parameter on connection
string to fix this issue.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
Example:<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
</div>
<blockquote class="tr_bq" style="text-align: left;">
<span style="background: rgb(250, 250, 250); font-family: "courier new"; font-size: 10.5pt; line-height: 107%;">connUrl = </span><span style="background: rgb(250, 250, 250); color: #a31515; font-family: "courier new"; font-size: 10.5pt; line-height: 107%;">"jdbc:sqlserver://mscorpag.mscorp.com:3306;databaseName=mscorp;user=rathish;password=pwd;ApplicationIntent=ReadOnly;MultiSubnetFailover=True"</span><span style="background: rgb(250, 250, 250); font-family: "courier new"; font-size: 10.5pt; line-height: 107%;">;</span></blockquote>
<div class="MsoNormal" style="text-align: left;">
<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<b><br /></b>
<b>Scenario 3: Availability Group Listener name not resolving to IP
address</b><o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Error:</b> After failover in a multi-subnet environment, ping command
from client not resolving to new IP address of the listener and DNS entry of
the listener name shows IPs of both subnets.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Possible Causes:</b><o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<b><br /></b></div>
<div class="MsoNormal" style="background: white;">
This error generally occurs, when listener is created using
<b>Failover Cluster Manager</b>, rather than from <b>SSMS</b>.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Solution: </b>Set the value of <i>RegisterAllProvidersIP</i> value as 0. This require restart
of listener network name resource.</div>
<div class="MsoNormal" style="background: white;">
<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
Example:<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>PowerShell: </b><o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<b><br /></b></div>
<blockquote class="tr_bq">
Import-Module FailoverClusters<br />
Get-ClusterResource dbcluster |Set-ClusterParameter<br />
RegisterAllProvidersIP 0</blockquote>
<div class="MsoNormal" style="background: white;">
<b>Cluster.exe: </b><o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<b><br /></b></div>
<blockquote class="tr_bq">
cluster /cluster: dbcluster res <networknameresource>mscorpag /priv<br />
<o:p></o:p></networknameresource>RegisterAllProvidersIP=0</blockquote>
<div class="MsoNormal" style="background: white;">
<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
If still ping to listener returning wrong IP address, from application/client
system, open command prompt as administrator and run the<i> <b>ipconfig /flushdns</b>
</i>command.<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Scenario 4: Server cannot host the availability group
listener IP address:<o:p></o:p></b></div>
<div class="MsoNormal" style="background: white;">
<b><br /></b></div>
<div class="MsoNormal" style="background: white;">
<b>Error:</b> Error 19456, Severity 16:<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Message:</b></div>
<div class="MsoNormal" style="background: white;">
<b><br /></b></div>
<blockquote class="tr_bq">
<i>Error 19456, Severity 16: None of the IP addresses configured for
the availability group listener can be hosted by the server '%.*ls'. Either
configure a public cluster network on which one of the specified IP addresses
can be hosted, or add another listener IP address which can be hosted.</i></blockquote>
<div class="MsoNormal" style="background: white;">
<b>Solution: </b>Add a new IP address manually to existing listener with different
subnet.</div>
<div class="MsoNormal" style="background: white;">
<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Scenario 5: Active Directory Policy</b><o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<b><br /></b></div>
<div class="MsoNormal" style="background: white;">
<b>Error:</b> Error 8557</div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Message:</b><br />
<b><br /></b></div>
<blockquote class="tr_bq">
<i>Error 8557 (Your computer could not be joined to the
domain. You have exceeded the maximum number of computer accounts you are
allowed to create in this domain. Contact your system administrator to have
this limit reset or increased.)</i></blockquote>
<div class="MsoNormal" style="background: white;">
A computer object is created automatically, whenever you create an
availability group listener. Even if you drop the listener, the computer object
remains there in Active Directory. By default, authenticated domain users can
create up to 10 computer objects, when this limit exceeds, it will through an
error at the time, availability group listener creation. <o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<div class="MsoNormal" style="background: white;">
<b>Solution: </b>You may contact domain admin to clean up this computer objects or ask
them to increase the default limit (I think it is a global variable).</div>
<div class="MsoNormal" style="background: white;">
<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<br /></div>
<h3 style="background: white;">
<b>Best practices:</b></h3>
<div class="MsoNormal" style="background: white;">
<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
</div>
<ul>
<li>Use static IP address for reliable connections and multi subnet
failover</li>
<li>Use meaningful and unique listener name for each availability
groups</li>
<li>Test it on staging server, before implementing it on production
environment</li>
<li>Additionally, to avoid the accidental deletion on cluster objects,
enable the Protect objects from accidental deletion option for Organizational Unit (OU).</li>
</ul>
<div class="MsoNormal" style="background: white;">
<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
<o:p></o:p></div>
<div class="MsoNormal" style="background: white;">
</div>
<div class="MsoNormal" style="background: white;">
<o:p></o:p></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2dw-d-frvzKCDkn0525gsXHyobcdHCNqiVMz0UOgLs-jwvSVpB31t8TLUHLeAfFBXldQmBeh-IC4MiJAbafctHKLrjMh8MZrq9h8JIAN2nwJ4lWj-nF7aRXMSs11LfNVhx__2viSWAIi_/s1600/11.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Protect object from accidental deletion" border="0" data-original-height="535" data-original-width="752" height="454" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2dw-d-frvzKCDkn0525gsXHyobcdHCNqiVMz0UOgLs-jwvSVpB31t8TLUHLeAfFBXldQmBeh-IC4MiJAbafctHKLrjMh8MZrq9h8JIAN2nwJ4lWj-nF7aRXMSs11LfNVhx__2viSWAIi_/s640/11.PNG" title="Protect object from accidental deletion" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Protect object from accidental deletion</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="background: white;">
<br />
I hope this article is helpful to you, if you have any queries or want
to share your thoughts on availability group listener, please write it in comment section.
Thank you!</div>
</div>
</div>
</div>
Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com38tag:blogger.com,1999:blog-2027709621250502166.post-38102450815930162502019-06-01T17:40:00.002+05:302021-03-06T13:49:51.211+05:30Step by step guide to add SSISDB to SQL Server 2016 Always On Availability Group (AG)<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="text-align: justify;">Starting from SQL Server 2016, you can
add SSISDB as Availability Database in Always On Availability Group (AG). This enables
high availability and automated failover for SSISDB. This article provides step
by step guide to add SSISDB to Availability Group.</span><br />
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Make sure, you have installed SQL
Server Integration Services on both primary and secondary replicas.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span>
<span style="font-family: inherit; mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><i>Related articles in Always On Availability Group (AG):</i></span><br />
<div style="text-align: left;">
<span style="font-family: inherit; mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><i><i style="font-family: inherit;"><i><span style="color: blue; font-family: inherit;"><a href="https://www.rathishkumar.in/2019/05/understanding-and-configuring-quorum.html" target="_blank">Understanding and configuring Windows Server Failover Cluster Quorum for SQL Server Always On</a></span></i></i></i></span></div>
<i style="font-family: inherit; text-align: left;"><a href="https://www.rathishkumar.in/2019/05/How-to-configure-Read-Only-routing-on-SQL-Server-2016-Always-On-Availability-Group-AG.html" target="_blank"><i><span style="color: blue; font-family: inherit;">How to configure Read-Only routing on SQL Server 2016 Always On Availability Group (AG)?</span></i></a></i></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b><br /></b></span>
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Step 1: Create SSIDB on primary
replica:</b><o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Right-click on the <b style="mso-bidi-font-weight: normal;">Integration Services Catalogs </b>and click
on <b style="mso-bidi-font-weight: normal;">Create Catalog Wizard</b> option</span></div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwht0E_SYyuF1UQnQ8BVeJ_2JN3-ifQFNVsHQZ8nVshYudDVU0PklTWnimSwcqCglfpmFR_ctIKLat-MqgYYGK7P4Pc5rE1h8X9SDigzOb5bM7vX5wSw-tYv-KRdt3Wu4alFUfqb7Rm0qu/s1600/1.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Creating Integration Service (SSIS) Catalog" border="0" data-original-height="208" data-original-width="447" height="296" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwht0E_SYyuF1UQnQ8BVeJ_2JN3-ifQFNVsHQZ8nVshYudDVU0PklTWnimSwcqCglfpmFR_ctIKLat-MqgYYGK7P4Pc5rE1h8X9SDigzOb5bM7vX5wSw-tYv-KRdt3Wu4alFUfqb7Rm0qu/s640/1.PNG" title="Creating Integration Service (SSIS) Catalog" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Creating Integration Service (SSIS) Catalog</b></td></tr>
</tbody></table>
<br />
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b><span><a name='more'></a></span>Step 2: Configure Integration Service Catalog:</b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b><br /></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">On <b style="mso-bidi-font-weight: normal;">Catalog Creation Wizard </b>select the <b style="mso-bidi-font-weight: normal;">Enable CLR integration</b> and <b style="mso-bidi-font-weight: normal;">Enable
automatic execution of Integration Services stored procedure at SQL Server
startup</b>. Then enter the password on <b style="mso-bidi-font-weight: normal;">Password</b>
and <b style="mso-bidi-font-weight: normal;">Retype Passwor</b>d text boxes and
click <b style="mso-bidi-font-weight: normal;">OK </b>button.</span></div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyDelNZYvAc5niWWRGg3Vc6MbRNTSy3x8ozOhv4uPakuIb8nqbho3UaH43QHmQC-5JXwU-YGrk3X9AURlEp2azhDrXr0C0ylQW161gRhgHwo_yHSFFXorFIyKvu5Hwz6yRHubDg4JpTJRd/s1600/2.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Configuring Integration Service Catalog" border="0" data-original-height="893" data-original-width="837" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgyDelNZYvAc5niWWRGg3Vc6MbRNTSy3x8ozOhv4uPakuIb8nqbho3UaH43QHmQC-5JXwU-YGrk3X9AURlEp2azhDrXr0C0ylQW161gRhgHwo_yHSFFXorFIyKvu5Hwz6yRHubDg4JpTJRd/s640/2.PNG" title="Configuring Integration Service Catalog" width="598" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td class="tr-caption" style="font-size: 12.8px;"><div style="text-align: center;">
<b>Configuring Integration Service Catalog</b></div>
</td></tr>
</tbody></table>
</td></tr>
</tbody></table>
<div class="MsoNormal" style="text-align: justify;">
<b><br /></b>
<b>Step 3: Adding SSISDB to Availability Group: </b><o:p></o:p></div>
<br />
<div class="MsoNormal" style="text-align: justify;">
On primary replica, add SSISDB to an
Availability Group (AG). Expand the <b>Always
On High Availability</b> and then <b>Availability
Groups</b>. Right-click on <b>Availability
Group</b> and click on <b>Add Database</b>
option.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-qwfetjomwBV2Zl8QhTqD-rmXWpmzwkpISqlbuJvGesqpwFkxObCik6rM0rU_J334AWARDq2wkXOFZNwwjDERqid4rqMAhSCtdlx-ZBtk8g_-0sQRQ4QDhOfHk-E4y7j3uLaj5s8XKtLu/s1600/3.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Adding SSISDB to Availability Group" border="0" data-original-height="315" data-original-width="295" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-qwfetjomwBV2Zl8QhTqD-rmXWpmzwkpISqlbuJvGesqpwFkxObCik6rM0rU_J334AWARDq2wkXOFZNwwjDERqid4rqMAhSCtdlx-ZBtk8g_-0sQRQ4QDhOfHk-E4y7j3uLaj5s8XKtLu/s640/3.PNG" title="Adding SSISDB to Availability Grou" width="599" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b style="font-size: 12.8px;">Adding SSISDB to Availability Group (AG)</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="text-align: justify;">
<b><br /></b>
<b>Step 4: Select database and decryption password:<o:p></o:p></b></div>
<br />
<div class="MsoNormal" style="text-align: justify;">
On <b>Add Database to Availability Group</b> wizard, click <b>Next </b>button. All the eligible databases
will be displayed here. Against <b>SSISDB</b>
you will see <b>Password required</b>. Enter
the password on <b>Password</b> column and
click on <b>Refresh</b> button.</div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnC51qjepaRc_7BpvFKUg36cBQPnnmfRZjH__lg2gQpaOcKcCagb368uHICISF_EJzucjXgyBusb8COmo7iE6h6MUzJHyirjJXd6zxzyR2oChbamFcubZ3lMGncyOYp7XPrStQ6xFcNR1K/s1600/5.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Always On - SSISDB prerequisites" border="0" data-original-height="29" data-original-width="570" height="32" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnC51qjepaRc_7BpvFKUg36cBQPnnmfRZjH__lg2gQpaOcKcCagb368uHICISF_EJzucjXgyBusb8COmo7iE6h6MUzJHyirjJXd6zxzyR2oChbamFcubZ3lMGncyOYp7XPrStQ6xFcNR1K/s640/5.PNG" title="Always On - SSISDB prerequisites" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Always On - SSISDB prerequisites</b></td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Now you will see the <b>Meets prerequisites</b> on status column.
Click on the <b>Next</b> button.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTrwVUTgwXcIO5MUpY3pe8rzy1sq4EY0-8jeIwhMuke6sx7h6eU-Ega83yQPQFrn0eNeuF3W3nUaNFPP3DuYbHZX-fItS1xdGzyCBoIulDzdUuWX3ZS8GJY2v3oghuLF3XzZqVo05j1CwJ/s1600/4.PNG" style="margin-left: auto; margin-right: auto;"><img alt="SSISDB Selection for AG and decryption password" border="0" data-original-height="40" data-original-width="573" height="44" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTrwVUTgwXcIO5MUpY3pe8rzy1sq4EY0-8jeIwhMuke6sx7h6eU-Ega83yQPQFrn0eNeuF3W3nUaNFPP3DuYbHZX-fItS1xdGzyCBoIulDzdUuWX3ZS8GJY2v3oghuLF3XzZqVo05j1CwJ/s640/4.PNG" title="SSISDB Selection for AG and decryption password" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>SSISDB Selection for AG and decryption password</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="text-align: justify;">
<b><br />Step 5: Selecting Secondary Replicas:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
On <b>Connect to Existing Secondary Replicas</b> page, you can either connect
to selective secondary replicas by clicking on <b>Connect</b> button against each replica name or you can connect to all secondary
replicas by clicking on <b>Connect All</b>
button.</div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
You will be prompted to enter username and password, now connected to secondary replica as mentioned user. Click on the <b>Next</b> button.</div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div style="text-align: justify;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaKOM5IBJVMnk-OAB9KRDaxlNeiFEcuE7JkRft_uGqd1l3FYvz5z4ufqaS1CQSGWG4ow7vPzg9k_LkZVvnVF4RYSbCyRBAbxJBiAAFWTWQTLoOkxSGCn1UkbChmPwTHMjambQ4MnnIP18g/s1600/6.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Choosing secondary replicas for SSIS Availability Database" border="0" data-original-height="679" data-original-width="822" height="528" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaKOM5IBJVMnk-OAB9KRDaxlNeiFEcuE7JkRft_uGqd1l3FYvz5z4ufqaS1CQSGWG4ow7vPzg9k_LkZVvnVF4RYSbCyRBAbxJBiAAFWTWQTLoOkxSGCn1UkbChmPwTHMjambQ4MnnIP18g/s640/6.PNG" title="Choosing secondary replicas for SSIS Availability Database" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Choosing secondary replicas for SSIS Availability Database</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Step 6: Initial Data Synchronization:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
On <b>Select Initial Data Synchronization</b> page, <b>Select your data synchronization preference</b>. Since, my <b>SSISDB</b> size is very small and my
cluster nodes are in same data centre, I am selecting <b>Automatic seeding</b>. You may use <b>Full
database and log backup</b> option or use other options as your requirement and
click <b>Next.</b></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg98RENQZGOfhTl2pU7a0xPkSi24ih37SC7xnFJrwWzRfDVfqID6hqZyyarJyI1-0Gj4TwjpMe8nf2S-WHi0cl30FTcDaxOuGR4U8EEJROqosrBfc5Dks0mQ2MSVJ87CuZruwxMuzo9r6jg/s1600/7.PNG" style="margin-left: auto; margin-right: auto;"><img alt="SSISDB - Always On - Initial Data Synchronization" border="0" data-original-height="688" data-original-width="809" height="544" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg98RENQZGOfhTl2pU7a0xPkSi24ih37SC7xnFJrwWzRfDVfqID6hqZyyarJyI1-0Gj4TwjpMe8nf2S-WHi0cl30FTcDaxOuGR4U8EEJROqosrBfc5Dks0mQ2MSVJ87CuZruwxMuzo9r6jg/s640/7.PNG" title="SSISDB - Always On - Initial Data Synchronization" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>SSISDB - Always On - Initial Data Synchronization</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="text-align: justify;">
<b><br /></b>
<b>Step 7: Validation:</b></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<div class="MsoNormal" style="text-align: justify;">
<br />
Validate and fix the failed
requirements and click on Next button.</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdZfSmObrU_ENUfMfWmRsU8X1GP9701qtFOn8Ux_OKLDKJKUN2SCgjFkKa2djwQajdgwApk57gcmytxVOcQt6daPIAT_rvi_WD3_oamOwk7vp3OgKnNY5t933DIPF9TvDF6wy8IAWJKJtv/s1600/8.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Adding SSISDB to Availability Group (AG) - Validation" border="0" data-original-height="683" data-original-width="814" height="536" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdZfSmObrU_ENUfMfWmRsU8X1GP9701qtFOn8Ux_OKLDKJKUN2SCgjFkKa2djwQajdgwApk57gcmytxVOcQt6daPIAT_rvi_WD3_oamOwk7vp3OgKnNY5t933DIPF9TvDF6wy8IAWJKJtv/s640/8.PNG" title="Adding SSISDB to Availability Group (AG) - Validation" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Adding SSISDB to Availability Group (AG) - Validation</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Step 8: Summary:</b></div>
<div class="MsoNormal" style="text-align: justify;">
<b><o:p></o:p></b></div>
<br />
<div class="MsoNormal" style="text-align: justify;">
Verify and confirm the choices made in
the adding <b>SSISDB</b> to <b>Availability Group</b> wizard and click on <b>Finish</b> button to continue.</div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiml3TLT3LQFtWqz9x5baHM5F4gt0ccfWSqpfPag7CVtHqKh_63_f6ZWoCqDeDtrzO00elNPnLWWs1I5vtg0XXrxx1sq-NcqKSw2_CKip6_JGSaoo16-HQP32OTHyHKW8qdIWzwM8X18vMT/s1600/9.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Adding SSISDB to Availability Group (AG) - Summary" border="0" data-original-height="676" data-original-width="816" height="530" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiml3TLT3LQFtWqz9x5baHM5F4gt0ccfWSqpfPag7CVtHqKh_63_f6ZWoCqDeDtrzO00elNPnLWWs1I5vtg0XXrxx1sq-NcqKSw2_CKip6_JGSaoo16-HQP32OTHyHKW8qdIWzwM8X18vMT/s640/9.PNG" title="Adding SSISDB to Availability Group (AG) - Summary" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><div style="text-align: center;">
<b style="font-size: 12.8px;">Adding SSISDB to Availability Group (AG) - Summary</b></div>
</td></tr>
</tbody></table>
<div style="text-align: center;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Step 9: Results:<o:p></o:p></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">The <b>Results</b> page provide the status of
adding SSISDB to Availability Group. Click on <b>Close</b> to exit the wizard.</span></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEha-M0p0ovFkg_atWFpAhBZ30ZxQczOrd4sKZeSteWI2DsjzW4ZHUQsiArf6XGj74w5FNm-FUzjQL47oejoFDBNhOxbKyiy3gJaB8JjOK683oywg2T5J3YDCc5RmdFqyrDoAmxW6vmdDJiq/s1600/10.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Adding SSISDB to Availability Group (AG) - Results" border="0" data-original-height="690" data-original-width="814" height="542" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEha-M0p0ovFkg_atWFpAhBZ30ZxQczOrd4sKZeSteWI2DsjzW4ZHUQsiArf6XGj74w5FNm-FUzjQL47oejoFDBNhOxbKyiy3gJaB8JjOK683oywg2T5J3YDCc5RmdFqyrDoAmxW6vmdDJiq/s640/10.PNG" title="Adding SSISDB to Availability Group (AG) - Results" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td class="tr-caption" style="font-size: 12.8px;"><div style="text-align: center;">
<span style="font-size: 12.8px;"><b>Adding SSISDB to Availability Group (AG) - Results</b></span></div>
</td></tr>
</tbody></table>
<div style="font-size: medium; text-align: left;">
</div>
</td></tr>
</tbody></table>
<b style="text-align: justify;"><br /></b>
<b style="text-align: justify;">Step 10:
Enabling Always On Support:</b><br />
<div class="MsoNormal" style="tab-stops: 185.25pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="tab-stops: 185.25pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Right-click
on the <b style="mso-bidi-font-weight: normal;">Integration Service Catalogs</b>
and click on <b style="mso-bidi-font-weight: normal;">Enable Always On Support</b>
option.<span style="mso-tab-count: 1;"> </span></span></div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsdjm7O0oN1uAryJzrVer21s3YA2dkbAn5iJ33qr61AHT_ylJZfk_mFttW8BfAUfZOdx01xSKzpvVa14uVtPvqvdQCFudPWcE9Wz9VWRonbEOwTMglUisIDYOkn5Awa_fFflvRUmjHa0Zb/s1600/14.png" style="margin-left: auto; margin-right: auto;"><img alt="Enabling Always On Support for Integration Service Catalog" border="0" data-original-height="206" data-original-width="457" height="288" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsdjm7O0oN1uAryJzrVer21s3YA2dkbAn5iJ33qr61AHT_ylJZfk_mFttW8BfAUfZOdx01xSKzpvVa14uVtPvqvdQCFudPWcE9Wz9VWRonbEOwTMglUisIDYOkn5Awa_fFflvRUmjHa0Zb/s640/14.png" title="Enabling Always On Support for Integration Service Catalog" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Enabling Always On Support for Integration Service Catalog</b></td></tr>
</tbody></table>
<br />
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Step 11: Connecting to secondary
replicas:<o:p></o:p></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />On <b style="mso-bidi-font-weight: normal;">Enable Support For AlwaysOn</b> wizard, click <b style="mso-bidi-font-weight: normal;">Connect All</b> to connect to all secondary replicas and click <b style="mso-bidi-font-weight: normal;">OK</b> button to close the wizard.</span></div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKa4lzawvaiMswQ7gHE-qp6-jeK440R4KDZu4k7kak0_y5sKbgc6PgirMQHirBHL3Z5a79188OhKW_5jnGY_aNYqw90H8TbjE2m5rFmVf2aCeevl9fPjf9eSCOgEIhDjjBdQoo81at2mI_/s1600/12.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Enable Support for Always On - Connecting secondary replicas" border="0" data-original-height="138" data-original-width="520" height="168" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKa4lzawvaiMswQ7gHE-qp6-jeK440R4KDZu4k7kak0_y5sKbgc6PgirMQHirBHL3Z5a79188OhKW_5jnGY_aNYqw90H8TbjE2m5rFmVf2aCeevl9fPjf9eSCOgEIhDjjBdQoo81at2mI_/s640/12.PNG" title="Enable Support for Always On - Connecting secondary replicas" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Enable Support for Always On - Connecting secondary replicas</b></td></tr>
</tbody></table>
<div style="text-align: justify;">
<b><br /></b>
<b>Step 12: Verifying Always On Support:</b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Post completing the previous step, you
can verify the Always On support by checking the SQL Server Agent jobs. The
following 2 jobs are created for <b>Failover Monitor</b> and <b>Server Maintenance </b>on
both primary and secondary replicas.</span></div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbvkRSjfOcGDFV0ipxYshT6sqqXtLYRSSxdlMPaWMsvmbZnJ2u14aIpFZD-yEOqVgE6F68yi-RL_lcCzeKTZB3JHZBAciwkoaTPrk8FySdD1ly4XlFbFSo19MxsDTe0ECy2UQOal4i5OaR/s1600/13.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Verifying Always On Support for Integration Service Catalog" border="0" data-original-height="112" data-original-width="263" height="272" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbvkRSjfOcGDFV0ipxYshT6sqqXtLYRSSxdlMPaWMsvmbZnJ2u14aIpFZD-yEOqVgE6F68yi-RL_lcCzeKTZB3JHZBAciwkoaTPrk8FySdD1ly4XlFbFSo19MxsDTe0ECy2UQOal4i5OaR/s640/13.PNG" title="Verifying Always On Support for Integration Service Catalog" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Verifying Always On Support for Integration Service Catalog</b></td></tr>
</tbody></table>
<div style="text-align: justify;">
<b><br /></b>
<b>Step 13: Monitoring SSISDB Availability
Database Health:</b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Open the <b style="mso-bidi-font-weight: normal;">Availability Group Dashboard</b> and check the status of the SSISDB and
Availability Group Database <b style="mso-bidi-font-weight: normal;">Synchronization
status</b>.<o:p></o:p></span></div>
<div style="text-align: justify;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJvTX_SJNHVMNLOQBSXdwd_xokhkzvg5LQl7AZpzBE4-JJuCbO4t4Kz6OMeFbN5_NAuq1EdEcqcOLdLoSRg-QA22ZcIdpw1CXhNjn8QF49MWuHcJLWm3Bbvdtp2D_QTAUYluX__7v3CoqN/s1600/11.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Monitoring SSIDB Availability Database Health" border="0" data-original-height="24" data-original-width="708" height="20" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJvTX_SJNHVMNLOQBSXdwd_xokhkzvg5LQl7AZpzBE4-JJuCbO4t4Kz6OMeFbN5_NAuq1EdEcqcOLdLoSRg-QA22ZcIdpw1CXhNjn8QF49MWuHcJLWm3Bbvdtp2D_QTAUYluX__7v3CoqN/s640/11.PNG" title="Monitoring SSIDB Availability Database Health" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Monitoring SSIDB Availability Database Health</b></td></tr>
</tbody></table>
<br />
<div class="MsoNormal" style="tab-stops: 185.25pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Great! You
have successfully added SSISDB as Availability Database on an Availability
Group (AG).<o:p></o:p></span></div>
<div style="text-align: justify;">
Note: When you are upgrading / patching, remove the SSISDB from Availability Group, apply the patches and add it back to Availability Group as mentioned <a href="https://www.rathishkumar.in/2019/06/step-by-step-guide-to-add-ssidb-to-alwayson-ag.html" target="_blank">here</a>. SSISDB in Always On Availability Group rule checks the SSISDB upgradation or patching status. I hope this article helps you. Please share your comments below and if you have difficulty in following any of the above steps, write in comment section, I will get back to you as soon as possible.</div>
</div>
Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com25tag:blogger.com,1999:blog-2027709621250502166.post-17872614052269652972019-05-25T14:39:00.003+05:302021-03-06T13:53:18.277+05:30Understanding and configuring Windows Server Failover Cluster Quorum for SQL Server Always On<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>What is quorum?<o:p></o:p></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">SQL Server availability solutions such
as Always On Availability Group (AG) and Failover Cluster Instance (FCI)
depends on Windows Server Failover Cluster (WSFC) for high availability and
failover operations. WSFC failover works based on the quorum. Generally, quorum
is minimum requirement of something to be up and running. In Windows Cluster,
quorum decides, minimum number of nodes required for cluster to be up and
running.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">In Windows Server cluster, more than
half of the nodes required for cluster to be up and run. For example, if you
have 4 node cluster, you need (4 / 2) + 1 nodes for cluster to work. <span style="mso-spacerun: yes;"> </span>This is called quorum in WSFC. <o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span>
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><i>Related articles in Always On:</i></span><br />
<ul>
<li><i><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><i><a href="https://www.rathishkumar.in/2019/06/step-by-step-guide-to-add-ssidb-to-alwayson-ag.html" target="_blank">Step by step guide to add SSISDB to SQL Server 2016 Always On Availability Group (AG)</a></i></span></i></li>
<li><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><i><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><i><a href="https://www.rathishkumar.in/2019/05/How-to-configure-Read-Only-routing-on-SQL-Server-2016-Always-On-Availability-Group-AG.html" target="_blank">How to configure Read-Only routing on SQL Server 2016 Always On Availability Group (AG)?</a></i></span></i></span></li>
</ul>
</div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Quorum and Roles:<o:p></o:p></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Windows Server designed in a way,
quorum understands the node which is active and nodes which are in stand-by. Also,
at a given point of time, there cannot be more than active nodes in a Windows
Server Failover Cluster. The quorum aware of this active node and in case of
fail-over, it will decide the next active node and stand-by nodes. In AG and FCI,
active means, nodes which owns the Resource Group at a given point of time.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Quorum and Partitioned Clusters:</b><o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
In Windows Server Failover Cluster,
each node communicates to other nodes in the cluster through a dedicated
network connection. When there is communication failure between nodes, there
will be partition of nodes and each node will think, other nodes are down and
trying to host Resource Group (active) for keeping the system up and running. In
Windows Server Failover Cluster (WSFC), more than one node cannot be active at
a given point of time. This conflict is called as<b> split-brain scenario</b>.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Quorum designed in a way, it is aware
of this communication, and if there are partitions in cluster, due to network
failure or some other issues, it will intervene to prevent the split-brain
scenarios. The partition with majority quorum will own the resource group and
cluster will force stop the cluster service from nodes of other partitions and
its removed from the WSFC.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">WSFC designed as when the
communication between removed nodes established either manually or
automatically, nodes can communicate to current cluster nodes, they will
automatically join the cluster and start their cluster service.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>How Windows Server Failover Cluster Quorum
works:<o:p></o:p></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Let’s assume, the cluster is
partitioned due to network failure, each partitioned nodes will try to own the
resource group. In order to achieve this, each subset of nodes has to prove the
quorum majority.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">For example, you have 5 node cluster,
post partition each subset will try to prove the quorum majority and subset
with 3 nodes will own the resources. Here, cluster quorum used to avoid the
split-brain situation. <o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">This works well, when you have odd
number of nodes, what happens when you have even number of nodes? For example,
you have a cluster with 4 nodes, and if network failure, you have partitioned
cluster of 2 subsets with 2 nodes each. It is 50/50 quorum, and again both
subset will consider itself as majority and try to own the resources, and fail
to prove the majority hence cluster will be down.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">How the cluster manages quorum, when
you have even number of nodes? There are 2 options:<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Add a cluster witness and increase the
vote count by one or decrease the vote of node as zero and make total number of votes as odd number. </b><o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Starting from Windows Server 2012
quorum is configured automatically using Dynamic Quorum. However cluster witness
must be added manually to cluster.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Dynamic Quorum:<o:p></o:p></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Post verifying the quorum majority,
the new majority definition will be updated among new cluster nodes. The concept
of the total number of votes adopted after a successive failure is known as Dynamic
Quorum. This allows cluster to lose one node, then another, then another until the
last standing member and majority definition is updated on each loss of nodes
dynamically.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">For example, you have a 3 node cluster
and if there is a failure, the subset of 2 nodes will survive and other node
will be removed from cluster. The quorum needed for cluster to be up and
running is more than half, in our case, we have left with only 2 nodes, the
quorum is 50/50. Now cluster will automatically zero down vote of either one of
the nodes and other node will be assigned as majority node with total vote of 1
out 1. This is called dynamic quorum.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Dynamic Witness:<o:p></o:p></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Starting from Windows Server 2012 R2,
the vote of cluster witness is calculated dynamically, when you have odd number
nodes, the witness does not have vote and, if there are even number of nodes,
the witness will have votes make total number of vote as odd. The process of
dynamically deciding the witness vote is called Dynamic Witness. <o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">For example, you have a 5 node cluster
with a cluster witness. By default, each node and witness will have 1 vote
each. The total vote for quorum of this cluster is 5, in this case, the value of
witness vote is 0, since it is already an odd number. Now assume, there is a
failure and one node removed from cluster, the total vote value will be 4, now
the cluster will automatically assign the voting of witness by 1 to make the
total value of vote as odd number. This process is called, dynamic witness.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Quorum Witness Types:<o:p></o:p></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">The Windows Server Failover Cluster
(WSFC) supports the following 3 types of quorum witnesses:<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Disk Witness:<o:p></o:p></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">A small size of storage disk
(typically 1 GB) attached with the cluster. The quorum disk is
accessible from all nodes in the cluster and is highly available. Disk witness
contains the copy of the cluster database.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>File Share Witness:<o:p></o:p></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">A shared folder from an external
server, which is accessible from all the nodes in the cluster. Usually from
Windows File Share or a folder from Domain Controller. It should be reliable
and DBA should aware of any changes to the file share, either access related or
maintenance related. File Share witness maintains the clustering information in
a witness.log file, but does not store copy of the cluster database.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><o:p> </o:p></span> </div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Cloud Witness:<o:p></o:p></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Introduced on Windows Server 2016,
BLOB storage account in Azure, which is accessible to all the nodes in the
cluster. Cloud witness maintains the clustering information in a witness.log
file, but does not store copy of the cluster database.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Quorum Models:<o:p></o:p></b></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Choosing the correct quorum model is
very critical decision for your availability solution to work. As we discussed
earlier, Windows Server provides multiple combination of quorum models. Let’s
look at what works best for your availability solution.</span></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ul>
<li>Node Majority</li>
<li>Node & Disk Majority</li>
<li>Node & File Share Majority</li>
<li>Node & Cloud Storage Majority</li>
<li>Disk-only - No Majority.</li>
</ul>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
Let’s look at each models in detail. <o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Node Majority:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Quorum majority is calculated based on
the active number of nodes in the cluster. By default, each node will have 1
vote each. There is not witness configured here. Recommended quorum mode for
Availability Groups (AGs) and Failover Cluster Instances (FCIs) when there is
an odd number of nodes.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Node & Disk Majority:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Quorum majority vote is calculated as
number of active nodes in the cluster and shared disk cluster resource.
Connectivity by any node to disk resource count as an affirmative vote for the
disk. Size of the disk should be 512MB minimum and it should be excluded from
Anti-Virus scanning. The disk resource should
be able to failover as a stand-alone instance. This model is recommended for
Failover Cluster Instance but not recommended for Availability Group (AGs). <o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Node & File Share Majority:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
In this model, voting is based on the
active number of nodes in the cluster and a file share resource. By default,
each node will have a vote and file share will have 1 vote. As a best practice,
the file share resource should not be physically located on any node of the
cluster, in-case of loss of that node will result in loss of 2 votes. This
model is recommended for Availability Groups and Failover Cluster Instances,
when there are even number of nodes in the cluster.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Node & Cloud Storage Majority:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
In this model, voting is calculated
based on the active number of nodes in the cluster and Azure Blob Storage. By
default, each node will have 1 vote and Azure Blob Storage will have a vote. The
Cloud witness is available from Windows Server 2016 and previous versions can
continue to use other models. Cloud Storage witness act as data centre and will
provide reliable voting. This is recommended for Always On Availability Group
and Failover Cluster Instance, when there is an even number of nodes.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p> </o:p> </div>
<div class="MsoNormal" style="text-align: justify;">
<b>Disk-Only - No majority:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
In this model, there is no quorum
majority calculated, active node is determined by the shared disk cluster
resource. There are no votes for the nodes in the cluster, only single-vote of
the disk-resource is required to be online as primary. All nodes in the cluster
must have connectivity to disk to gain an affirmative vote and be online. This
model is not recommended for Always On Availability Group and Failover Cluster
Instances. This model exists only for backward compatibility.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<b><br /></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Example
Scenarios:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Scenario 1: Two nodes without witness:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Cluster will dynamically decide the
quorum, vote of either one of two nodes will be zeroed down and another node
will be assigned as total votes 1 out 1 and it has quorum majority and own the
resources.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
If the current active node fails
unexpectedly, then the cluster will go down, it is a single point of failure,
but if is a graceful shutdown, the voting will be shifted to the another node
and the cluster will be still up. There is a fifty percent chance, the cluster
will survive one failure. This is the reason we need a cluster witness. <o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Scenario 2: Two nodes with witness:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b><br /></b></div>
<div class="MsoNormal" style="text-align: justify;">
In this case, the quorum has total 3
votes. If any of a node or witness goes down, you will still have a node and
witness or 2 nodes with quorum majority and cluster will be up and running.
This cluster will survive maximum one failure. <o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Scenario 3: Three nodes without witness:</b><o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Total vote is 3, if any one of the
nodes is, the vote will be 2/3 and the cluster can survive, at this point,
dynamic quorum updated and cluster will be scenario 1. This cluster can survive
one failure and there is a fifty percent chance of next failure.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Scenario 4: Three nodes with witness:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
In this case, the witness does not
have vote (dynamic witness), so the total vote is an odd number 3. If any one
of the nodes fails, the cluster will be scenario 2. This cluster can survive 2
subsequent failures of node.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Scenario 5: Four nodes without witness:</b><o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
In this scenario, the cluster
automatically zero downs one vote from a node and make quorum majority as odd
number. If a node fails, the cluster becomes scenario 3. This cluster can
survive 2 subsequent failures of nodes and there is a fifty percent chance of
next subsequent failure of nodes.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Scenario 6: Four nodes with witness:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
All four nodes and witness will have a
vote and make the quorum majority an odd number. In case of failure, the cluster
will become scenario 4. The cluster will survive 3 node failures.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Configuring the failover cluster
quorum:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
There are two ways you can configure
the cluster quorum, Failover Cluster Manager and Failover Cluster Windows PowerShell
cmdlets. In this article, we are going to configure node and file share majority
for a cluster. The Failover Cluster Manager wizard is self-explanatory and
having understanding of the quorum concept will help you in configuring other
quorum models and it is very similar.<o:p></o:p></div>
<br />
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Go to <b>Failover Cluster Manager</b> and
connect to the cluster. Right click on the cluster name and go to <b>More Actions</b>
and choose the <b>Configure Cluster Quorum Settings</b>.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtzQq__VTHcxu9TVSBtmLPLNnqSDDbb-3_uWGVAFLazfGA4DUV0EMPzIskz-EO2ooP_uf36_H4RzdIGUoVLysPmzo6XWQhA_cc1X6KuZoEePz9fjmxmGQPoE5o59bhEfESC38vvpHKu8jo/s1600/1.png" style="margin-left: auto; margin-right: auto;"><img alt="Failover Cluster Manager - Quorum Configuration" border="0" data-original-height="592" data-original-width="801" height="473" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtzQq__VTHcxu9TVSBtmLPLNnqSDDbb-3_uWGVAFLazfGA4DUV0EMPzIskz-EO2ooP_uf36_H4RzdIGUoVLysPmzo6XWQhA_cc1X6KuZoEePz9fjmxmGQPoE5o59bhEfESC38vvpHKu8jo/s640/1.png" title="Failover Cluster Manager - Quorum Configuration" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Failover Cluster Manager - Quorum Configuration</b></td></tr>
</tbody></table>
<div style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b><span><a name='more'></a></span>Configure Quorum Cluster Wizard</b> will
be opened, under <b>Select Quorum Configuration Option</b> choose the <b>Select the quorum
witness Option.</b><o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixtu5-t4f7A3dg72G0ERKUSCA_c9qWKPtAtc3yGpx8ywu0ivR4n0ppM4WFgo4NAYJACDO8nK6lRpY730myu_JGMG8RTIr1CdZMom2ZyRgyAIG4myNg3wnLHsB5yaxo1SrlvGk6mgdVkcvC/s1600/2.png" style="margin-left: auto; margin-right: auto;"><img alt="Configure Cluster Quorum Wizard" border="0" data-original-height="472" data-original-width="682" height="442" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixtu5-t4f7A3dg72G0ERKUSCA_c9qWKPtAtc3yGpx8ywu0ivR4n0ppM4WFgo4NAYJACDO8nK6lRpY730myu_JGMG8RTIr1CdZMom2ZyRgyAIG4myNg3wnLHsB5yaxo1SrlvGk6mgdVkcvC/s640/2.png" title="Configure Cluster Quorum Wizard" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Configure Cluster Quorum Wizard</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="text-align: justify;">
Click <b>Next</b> and move to the <b>Select
Quorum Witness page</b> and select the <b>Configure a file share witness option </b>and
click on the <b>Next</b> button.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRwlEHVQ9Ry1rKJog6xw5O4j_8Dar6da4AOYGN1qSnJQ8Iv2Qielqp3DRrhGP4oF9J_7SKVftfiej87gOkV0Rp6ClPcK4LGsKX9NJFVqf_v58ekg1BU182ZYARcMCzd29oLtQLNc6GGEbX/s1600/3.png" style="margin-left: auto; margin-right: auto;"><img alt="Quorum Configuration - Select Quorum Witness" border="0" data-original-height="472" data-original-width="684" height="440" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRwlEHVQ9Ry1rKJog6xw5O4j_8Dar6da4AOYGN1qSnJQ8Iv2Qielqp3DRrhGP4oF9J_7SKVftfiej87gOkV0Rp6ClPcK4LGsKX9NJFVqf_v58ekg1BU182ZYARcMCzd29oLtQLNc6GGEbX/s640/3.png" title="Quorum Configuration - Select Quorum Witness" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Quorum Configuration - Select Quorum Witness</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="text-align: justify;">
<br />
Under <b>Configure File Share Witness</b>
page enter the <b>File Share Path</b> and click on <b>Next</b> button. The File Share Path is
nothing but a folder path, example:<b> \\SERVER\Folder\Cluster.<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjERYUJbnA-x2sGEBcx4GVUAexIPMyRVWNBWOmrSxLU7t3ifo4Wh1cyHU6V6pZnVqfUTc-zEKRhHH6JUqwWV3NZ17rwBk67bQMUsiAbbs3OLELrjfvgcaDGX4wHmCQx0K1XBtbAdwLFI8y_/s1600/4.png" style="margin-left: auto; margin-right: auto;"><img alt="Quorum Configuration - File Share Witness Configuration" border="0" data-original-height="474" data-original-width="686" height="442" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjERYUJbnA-x2sGEBcx4GVUAexIPMyRVWNBWOmrSxLU7t3ifo4Wh1cyHU6V6pZnVqfUTc-zEKRhHH6JUqwWV3NZ17rwBk67bQMUsiAbbs3OLELrjfvgcaDGX4wHmCQx0K1XBtbAdwLFI8y_/s640/4.png" title="Quorum Configuration - File Share Witness Configuration" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Quorum Configuration - File Share Witness Configuration</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="text-align: justify;">
<br />
On <b>Confirmation</b> page, review the
settings and click on <b>Next</b> button.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPTiVfbk-MokmZPiNF-GI1NQhGbF6Ev8QgdaKFVmaxbcktuvBymC8MllagjEM3MDvVDkYIk95MyuONhr8gZh4pbymH7L3myDrgXHhe-g8ouID7jsIDPvp1yYlopY2sFbofJuPrCckibj8j/s1600/5.png" style="margin-left: auto; margin-right: auto;"><img alt="Quorum Configuration - Confirmation Wizard" border="0" data-original-height="472" data-original-width="683" height="442" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPTiVfbk-MokmZPiNF-GI1NQhGbF6Ev8QgdaKFVmaxbcktuvBymC8MllagjEM3MDvVDkYIk95MyuONhr8gZh4pbymH7L3myDrgXHhe-g8ouID7jsIDPvp1yYlopY2sFbofJuPrCckibj8j/s640/5.png" title="Quorum Configuration - Confirmation Wizard" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Quorum Configuration - Confirmation Wizard</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="text-align: justify;">
<br />
<b>Summary</b> – You have successfully
configured the quorum settings for the cluster. Click on the <b>Finish</b> button to
close the wizard.<o:p></o:p></div>
<div style="text-align: justify;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6dY8O_kXYvMjMtN-Xqi7DkcautkM7GQ8qKbiF1NccGS7JJLvKV_-Ua54crlyyJxiTdx-6HTvKb9pt4TX5xPf67ld0YXlk1ZvA7hUBxgYqA3leUTjGRksHM5L0nfMw6F7TOOTd8JwXgsPA/s1600/6.png" style="margin-left: auto; margin-right: auto;"><img alt="Quorum Configuration - Summary Wizard" border="0" data-original-height="470" data-original-width="683" height="440" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6dY8O_kXYvMjMtN-Xqi7DkcautkM7GQ8qKbiF1NccGS7JJLvKV_-Ua54crlyyJxiTdx-6HTvKb9pt4TX5xPf67ld0YXlk1ZvA7hUBxgYqA3leUTjGRksHM5L0nfMw6F7TOOTd8JwXgsPA/s640/6.png" title="Quorum Configuration - Summary Wizard" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Quorum Configuration - Summary Wizard</b></td></tr>
</tbody></table>
<div style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Viewing Cluster Quorum information:</b><o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>PowerShell:</b> In command window, type <b>Get-ClusterQuorum</b>
to view the cluster details:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjftgybWj2kF2jZkUACgGWvEajQyfHVE2EtRg1_ajFw5xRYwqafkw_ZZy7dipoxpbdjVS_Ho_Fn0IzEW17lWeMSRKt7QnbOfgL-6uZhCbeXEkV6nWQTlBJEI_5sDdNstGc6ZlyPYXgCi_Ez/s1600/7.png" style="margin-left: auto; margin-right: auto;"><img alt="View Cluster Quorum Information - Get-ClusterQuorum PowerShell cmdlet" border="0" data-original-height="109" data-original-width="586" height="118" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjftgybWj2kF2jZkUACgGWvEajQyfHVE2EtRg1_ajFw5xRYwqafkw_ZZy7dipoxpbdjVS_Ho_Fn0IzEW17lWeMSRKt7QnbOfgL-6uZhCbeXEkV6nWQTlBJEI_5sDdNstGc6ZlyPYXgCi_Ez/s640/7.png" title="View Cluster Quorum Information - Get-ClusterQuorum PowerShell cmdlet" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>View Cluster Quorum Information - Get-ClusterQuorum PowerShell cmdlet</b></td></tr>
</tbody></table>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<b>SSMS:</b> Right click on <b>Availability
Groups</b> -> Click on <b>Show Dashboard </b>-> In Dashboard – click on <b>Availability
Group</b> -> Click on <b>View Cluster Quorum Information</b> at the right side of the dashboard.<o:p></o:p></div>
<div style="text-align: justify;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi02L4JAfpe_os-37CmKWWJDwMRov4Au7ECF3_TVydEwSI9a-MlHmnfM-KgU5mR9mzWR_QGc_CWUUTnlUA3kHC5Q0fptq_hVB8srHo5FArVRPHMAz6Rw2kq0Y5kUmwyYl4pqHcxPPde7rIQ/s1600/8.png" style="margin-left: auto; margin-right: auto;"><img alt="View Cluster Quorum Information - Availability Group - Monitoring Dashboard - SSMS" border="0" data-original-height="577" data-original-width="1014" height="364" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi02L4JAfpe_os-37CmKWWJDwMRov4Au7ECF3_TVydEwSI9a-MlHmnfM-KgU5mR9mzWR_QGc_CWUUTnlUA3kHC5Q0fptq_hVB8srHo5FArVRPHMAz6Rw2kq0Y5kUmwyYl4pqHcxPPde7rIQ/s640/8.png" title="View Cluster Quorum Information - Availability Group - Monitoring Dashboard - SSMS" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>View Cluster Quorum Information - Availability Group - Monitoring Dashboard - SSMS</b></td></tr>
</tbody></table>
<div style="text-align: justify;">
<br /></div>
<br />
<div class="MsoNormal" style="text-align: justify;">
<b>TSQL:</b> Connect to cluster server and
run the below query:</div>
<br />
<blockquote class="tr_bq" style="text-align: justify;">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">SELECT</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;"><span style="mso-spacerun: yes;"> </span>member_name</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt;">,</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;"> member_state_desc</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt;">,</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt;"> number_of_quorum_votes<span style="mso-spacerun: yes;"><br /> </span><o:p></o:p></span><span style="color: blue; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">FROM</span><span style="color: black; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;"><span style="mso-spacerun: yes;"> </span></span><span style="color: lime; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">sys</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">.</span><span style="color: lime; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">dm_hadr_cluster_members</span><span style="color: grey; font-family: "consolas"; font-size: 9.5pt; line-height: 107%;">;<o:p></o:p></span></blockquote>
<br />
<div class="MsoNormal" style="text-align: justify;">
<b>Managing and configuring Node Weight
and Cluster Voting:</b><o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
To manage, node weight, go to
<b>Configure Cluster Quorum Wizard</b> (mentioned above) and click on the <b>Advanced
quorum configuration</b> and click <b>Next</b>.<o:p></o:p></div>
<div style="text-align: justify;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhPwFdZKumsaFoLqMwegSA1QzuRKYIvbpeIHyMhcvc6fGyaRBt7-k5Jiiqol-lw7mqoFgHWtk6klq7-W7FzkjR5YM1rpB0WmuU-hIO5_EAf-bqpVa0yUw_4NA6c0UkRlgEA-fEmPE1N3_R/s1600/9.png" style="margin-left: auto; margin-right: auto;"><img alt="Configure Cluster Quorum Wizard - Advanced quorum configuration" border="0" data-original-height="472" data-original-width="684" height="440" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhPwFdZKumsaFoLqMwegSA1QzuRKYIvbpeIHyMhcvc6fGyaRBt7-k5Jiiqol-lw7mqoFgHWtk6klq7-W7FzkjR5YM1rpB0WmuU-hIO5_EAf-bqpVa0yUw_4NA6c0UkRlgEA-fEmPE1N3_R/s640/9.png" title="Configure Cluster Quorum Wizard - Advanced quorum configuration" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Configure Cluster Quorum Wizard - Advanced quorum configuration</b></td></tr>
</tbody></table>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<span face=""calibri" , sans-serif" style="font-size: 11pt; line-height: 107%;">Under <b>Select Voting Configuration</b> page, click on
<b>Select Nodes</b> and uncheck the nodes, which you don’t want add weight and click
on <b>Next</b> button.</span></div>
<div style="text-align: justify;">
<span face=""calibri" , sans-serif" style="font-size: 11pt; line-height: 107%;"><br /></span></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmp3uM5WNTqo_u1Nvggf8sycp5rkDsvz20Gczk_lg2Zm33ulKlJCI9romQpN3ugBNP2yQFJ_UeTdY2GwzSypdJxa7VzbSjMBTmf1oTNkTOnqNFrWJjKY8tBi4rNnOmITa2h_poJI3N0rX8/s1600/10.png" style="margin-left: auto; margin-right: auto;"><img alt="Cluster Quorum Configuration - Managing Quorum voting and configuration" border="0" data-original-height="473" data-original-width="686" height="440" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmp3uM5WNTqo_u1Nvggf8sycp5rkDsvz20Gczk_lg2Zm33ulKlJCI9romQpN3ugBNP2yQFJ_UeTdY2GwzSypdJxa7VzbSjMBTmf1oTNkTOnqNFrWJjKY8tBi4rNnOmITa2h_poJI3N0rX8/s640/10.png" title="Cluster Quorum Configuration - Managing Quorum voting and configuration" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Cluster Quorum Configuration - Managing Quorum voting and configuration</b></td></tr>
</tbody></table>
<div style="text-align: justify;">
<span face=""calibri" , sans-serif" style="font-size: 11pt; line-height: 107%;"><br /></span></div>
<span style="line-height: 107%;"></span><br />
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Configure the remaining pages as
mentioned above and close the wizard. Now go to view cluster quorum information
section to verify the changes.<o:p></o:p></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Best practices for configuring quorum
vote:</b><o:p></o:p></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">When Windows Server Failover Cluster
nodes spread across multiple data centres, there is a possibility of network
latency or failure, and cluster may partition, if small disruptions. To prevent
this, configure votes to nodes, which can communicate without any issues under
normal circumstances.<o:p></o:p></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Configure votes to only nodes, which
are hosting SQL Server instances, you can skip other nodes, this will minimize
the failure chances.<o:p></o:p></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Configure votes to primary node and
most eligible stand-by nodes, which are going to be primary, in case of
failover.<o:p></o:p></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Try to maintain the total number of
votes as odd number, in-case of even nodes, add witness and leverage the
benefits of dynamic quorum and dynamic witness features.<o:p></o:p></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><b>Summary<o:p></o:p></b></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></div>
<div class="MsoNormal" style="font-family: Calibri, sans-serif; font-size: 11pt; text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Great! we have covered what is quorum
and its importance, new quorum features introduced in Windows Server 2012, witness’s
types and configurations and cloud witness features, quorum models and finally
managing and configuring windows failover cluster quorum, voting and best practices. Thanks for reading
the article till the end. I hope you enjoyed this post, start building your own
servers and test different quorum models and scenarios. Please write your
questions and views on the comment section. I will reply to you as quick as possible.
Thank you!</span></div>
</div>
Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com10tag:blogger.com,1999:blog-2027709621250502166.post-60480168685608250292019-05-17T21:22:00.001+05:302021-03-06T13:59:01.446+05:30How to configure Read-Only routing on SQL Server 2016 Always On Availability Group (AG)? <div dir="ltr" style="text-align: left;" trbidi="on"><div style="text-align: justify;">The great advantage of Always On Availability Group over other availability solutions is their ability to scale-out read operations (SELECT queries). Read-only routing is a feature of Always On Availability Group, which redirects connection requests from applications into readable secondary. In SQL Server 2016 Always On Availability Group, you can configure upto 8 readable secondary replicas.</div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">The connections are redirected based on the routing rules. Always On Availability Group provides the following options to define the rules:</div><div><div><ul style="text-align: left;"><li style="text-align: justify;">Read-Only Routing URL</li>
<li style="text-align: justify;">Read-Only Routing List</li>
</ul></div></div><div><div style="text-align: justify;">Before defining the routing rules, we must understand the following conditions:</div></div><div style="text-align: justify;"><br />
</div><div><div style="text-align: justify;"><span><a name='more'></a></span>The application must connect to the Virtual Network Name (VNN) and not to the secondary replica directly. VNN is defined at the time of configuring listener.</div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">The application connection string must contain the read-only connection parameter, <b>ApplicationIntent=ReadOnly</b>;</div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">There must be at least on readable secondary exist on the AG. Let’s configure the routing rules:</div></div><div style="text-align: justify;"><br />
</div><div><div style="text-align: justify;"><b>Read-only Routing URL:</b></div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">The URL is used when an application explicitly trying to connect to readable secondary with read-only intent. This URL contains the hostname and port number.<br />
<br />
</div><div style="text-align: justify;"><blockquote class="tr_bq"><b>Format<span style="white-space: pre;"> </span>: TCP://server.domain.com:1433</b><br />
<b>Example<span style="white-space: pre;"> </span>: TCP://node1.ms.com:1433 (note: this on node1.ms.com)</b></blockquote></div><div style="text-align: justify;">This rule is applicable only when the node is acting as a secondary replica (if it is primary, obviously it will accept, read and write connections).</div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;"><b>Configure read-only routing URL using T-SQL:</b><br />
<b><br />
</b> <br />
<blockquote class="tr_bq"><b>ALTER AVAILABILITY GROUP [TestAG]<br />
</b><b>MODIFY REPLICA ON<br />
</b><b>N’node1’<br />
</b><b>WITH<br />
</b><b>(SECONDARY_ROLE (READ_ONLY_ROUTING_URL = N’TCP://node1.ms.com:1433’));<br />
</b><b>GO</b></blockquote><br />
<div class="MsoNormal"><o:p></o:p></div></div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;"><b>Read-Only Routing List:</b></div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">The read-only routing list contains the list of readable secondary with their priority. For example, node1.ms.com is the primary replica, and when an application is trying to connect to AG with explicit read-only intent, the primary replica will redirect the read-only connection to available secondary replicas as defined on the routing-list.<br />
<br />
<blockquote class="tr_bq"><b>Format<span style="white-space: pre;"> </span>: ‘replica1’,’replica2’<br />
Example : ‘node2’,’node3’ (note; on node1.ms.com)</b></blockquote></div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;"><b>Configure read-only routing list using T-SQL:</b></div><div style="text-align: justify;"><br />
</div><blockquote class="tr_bq"><div style="text-align: justify;"><b>ALTER AVAILABILITY GROUP [TestAG] </b></div><div style="text-align: justify;"><b>MODIFY REPLICA ON </b></div><div style="text-align: justify;"><b>N’node1’ </b></div><div style="text-align: justify;"><b>WITH (PRIMARY_ROLE (READ_ONLY_ROUTING_LIST= (N’node2’, N’node3’))); </b></div><div style="text-align: justify;"><b>GO</b></div></blockquote><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">Note the role, it is when node1.ms.com server is in primary role, it will take effect. Now you have configured the routing URL and routing list, follow the below steps to verify it is working as expected.</div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;"><b>Steps to verify Read-Only routing:</b></div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">In SSMS login, enter server name as Always On AG listener and go to options and find the additional connector parameters.</div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">Enter the below connection parameters and click on the connect button.</div><div style="text-align: justify;"><br />
</div><blockquote class="tr_bq" style="text-align: justify;"><b>ApplicationIntent=ReadOnly; InitialCatalog=databasename;</b></blockquote><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">Open new query window and see the server name to identify, currently which server it is connecting.<br />
<br />
</div><div style="text-align: justify;"></div><blockquote class="tr_bq" style="text-align: justify;"><b>SELECT @@SERVERNAME;</b></blockquote><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">The output should be name of Read-only Secondary Replica. </div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">At the Read-Only Secondary Replica, if you try to update tables, you will get the error message stating, you cannot perform DML operation on secondary replica.</div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">If you are not getting the Read-Only Secondary replica, mention it on comment section, I will be glad to help you and get back to as soon as possible.</div><div style="text-align: justify;"></div><br />
<div style="text-align: justify;"><b>Load-balancing across Read-Only Secondary Replicas:</b></div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">Starting from SQL Server 2016, you can configure the load balancing across the read-only replicas. Load balancing can be configured as below:</div><div style="text-align: justify;"><br />
</div><blockquote class="tr_bq"><div style="text-align: justify;"><b>ALTER AVAILABILITY GROUP [TestAG] </b></div><div style="text-align: justify;"><b>MODIFY REPLICA ON </b></div><div style="text-align: justify;"><b>N’node1’ </b></div><div style="text-align: justify;"><b>WITH </b></div><div style="text-align: justify;"><b>(</b></div><div style="text-align: justify;"><b>PRIMARY_ROLE </b></div><div style="text-align: justify;"><b>(</b></div><div style="text-align: justify;"><b>READ_ONLY_ROUTING_LIST= ((’node1’, ’node2’,’node3’), ‘node4’, ‘node5’)); </b></div><div style="text-align: justify;"><b>));</b></div><div style="text-align: justify;"><b>GO</b></div></blockquote><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">Load-balancing performed using Round-robin algorithm, and connection requests are load-balanced between node1, node2 and node3. Next priority will be node4 and the last priority will be node5.</div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;">I hope this article helps you in configuring the read-only routing, if you have questions or doubts, please mention it on comment section. </div></div></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com10tag:blogger.com,1999:blog-2027709621250502166.post-13153824045525296042019-02-22T17:24:00.000+05:302019-02-24T14:05:43.421+05:30Architecting reliable backup and recovery strategy<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">I have been managing multiple databases, mostly in Microsoft SQL Server and MySQL server, both on on-premise and cloud. We have faced a lot of challenging issues such as records deleted from a user table, backup file is corrupted, backup file is not compatible, backup files got deleted, backup storage is full and backup is running for long time, etc. When you are facing this issues for the first time, it is surprising to see new kind of issues every day and if you are not good in documentation, repetitive issues will keep occurring and we keep fixing rather than suctioning. If you are facing same challenges, then you need to focus on your backup and recovery strategy. <o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
A well-designed backup and recovery strategy maximizes data availability and minimizes data loss without tolerating business requirement. In this post, we will discuss about the following topics:</div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ol style="text-align: left;">
<li style="text-align: justify;"><span style="text-align: left;">Recovery Time Objective (RTO)</span></li>
<li style="text-align: justify;"><span style="text-align: left;">Recovery Point Objective (RPO)</span></li>
<li style="text-align: justify;"><span style="text-align: left;">Business Strategy</span></li>
<li style="text-align: justify;">Backup </li>
<ul>
<li style="text-align: justify;">Backup Type</li>
<li style="text-align: justify;">Backup Frequency</li>
<li style="text-align: justify;">Backup Device</li>
<li style="text-align: justify;">Retention and Archival plans</li>
</ul>
<li style="text-align: justify;">Restore</li>
<ul>
<li style="text-align: justify;">Backup Availability</li>
<li style="text-align: justify;">Restoration Access</li>
<li style="text-align: justify;">Documentation</li>
</ul>
<li style="text-align: justify;">Testing</li>
</ol>
<div style="text-align: justify;">
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Recovery Time Objective (RTO): Time taken to recover the data<o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Recovery Time Objective defines, how long would an interruption to data access need to be, before the business ceased to be viable / capable of working successfully?<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Best Practice: <o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Backup and Restoration plan that involves quick recovery with small data loss might be acceptable to business than a plan that reduces data loss but takes much longer to implement.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Deciding optimal RTO might involve following things:</span></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ul>
<li><span style="text-indent: -18pt;"><span style="font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span><span style="text-indent: -18pt;">Identifying the correct backup media</span></li>
<li><span style="text-indent: -18pt;"><span style="font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span><span style="text-indent: -18pt;">Identifying the authorized persons to perform restore</span></li>
<li><span style="text-indent: -18pt;"><span style="font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span><span style="text-indent: -18pt;">Identifying documentation related to the restore</span></li>
</ul>
<b>Example:</b><br />
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">RTO for the Stock Exchange System might be 1 minutes – which means, in case of any failure on database system, backup should be restored and system should be online within a minute. <o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Recovery Point Objective (RPO): The amount of data loss<o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Recovery Point Objective defines how much data loss will be acceptable to business. <o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Best Practice:<o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Backup plan should be able to recover all committed data without any loss and it should be acceptable to loss the data which is in process at the time of failure.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Example:<o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">RPO for the banking system might be 1 seconds – which means, in case of any failure, the bank can loss at most 1 second worth of data. <o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Business Strategy:<o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Backup and recovery plan should be in sync with business requirement. As a best practice, it should be communicated with business stakeholders and expectations of the business users should be managed in line with the agreed strategy.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Each database must be categorized based on the importance to the core functions of the organization and RPO and RTO for each database must be documented well in advance and communicated to all stakeholders.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Example: <o:p></o:p></span></b></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-align: justify; text-autospace: none;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-align: justify; text-autospace: none;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">In certain business, Sales database is very critical than Marketing database and needs to restored in high priority, in case of failure. The RPO and RTO for Sales and Marketing databases will be defined accordingly.<o:p></o:p></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-align: justify; text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-align: justify; text-autospace: none;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">The business requirements will determine all aspects of the backup strategy, including:</span></div>
<div class="MsoListParagraphCxSpFirst" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-add-space: auto; mso-layout-grid-align: none; mso-list: l1 level1 lfo2; text-align: justify; text-autospace: none; text-indent: -18.0pt;">
</div>
<ul>
<li><span style="font-family: "symbol"; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;"><span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">How frequently backups need to occur?<o:p></o:p></span></li>
<li><span style="font-family: "symbol"; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;"><span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">How much data is to be backed up each time?<o:p></o:p></span></li>
<li><span style="font-family: "symbol"; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;"><span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">The type of media, that the backups will be held on?<o:p></o:p></span></li>
<li><span style="font-family: "symbol"; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;"><span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Retention and Archival plans for the media? </span></li>
</ul>
<b style="text-indent: -18pt;">Backup Type:</b><br />
<div class="MsoListParagraphCxSpFirst" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-add-space: auto; mso-layout-grid-align: none; mso-list: l1 level1 lfo2; text-align: justify; text-autospace: none; text-indent: -18.0pt;">
<b style="text-indent: -18pt;"><br />
</b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Backup and recovery plan should clearly define the backup type required for different scenarios. A full backup might be appropriate in certain cases and partial backup will be appropriate in other scenario.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">For example, Product database may have huge number of records and backup and restoration of a huge backup is time consuming. When you have RTO of 1 minutes, it is not right strategy to have a full backup which needs 1 hour of restoration time. Similarly, when RPO is defined as 5 seconds, we cannot run full backup on every 5 seconds.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Backup and Recovery strategy should address this question. It should choose the combination of different backup type to achieve the defined RPO and RTO metrics. <o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Generally, there are 3 types of backups are used:</span></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ul>
<li><span style="text-indent: -18pt;"><span style="font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span><span style="text-indent: -18pt;">Full backup.</span></li>
<li><span style="text-indent: -18pt;"><span style="font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span><span style="text-indent: -18pt;">Differential backup.</span><span style="text-indent: -18pt;"><span style="font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span></li>
<li><span style="text-indent: -18pt;"><span style="font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span><span style="text-indent: -18pt;">Log backup.</span></li>
</ul>
<b>Backup Frequency:</b><br />
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Most of the places, the backups are implemented in below frequency:</span></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ul>
<li><span style="text-indent: -18pt;"><span style="font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span><span style="text-indent: -18pt;">Full backup – every week</span></li>
<li><span style="text-indent: -18pt;"><span style="font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span><span style="text-indent: -18pt;">Differential backup – every day</span></li>
<li><span style="text-indent: -18pt;"><span style="font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span><span style="text-indent: -18pt;">Log backup – every hour</span></li>
</ul>
<br />
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Review the above schedule and think, will it satisfy the business requirement. There is no right or wrong questions, the above frequency might satisfy the RPO of 1 hours and depends on the size of the database and where backup stored, RTO might be achievable. There is no one size fit all approach here, you have to understand the business requirement and define the RTO and RPO. <o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">When business demand no data loss, log backup of every 1 hour will not help, in case any failure, with the above backup plan, you may lose the data worth 1 hour. A well-designed backup and restore plan will address this issue and it should be documented and shared across business stakeholders.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<!-- Google Large Ad Starts --><br />
<script async="" src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<ins class="adsbygoogle" data-ad-client="ca-pub-9474984742004767" data-ad-format="fluid" data-ad-layout="in-article" data-ad-slot="6514005873" style="display: block; text-align: center;"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Backup Device:</span></b><br />
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">This is one of the important factors, choosing the wrong backup device, will negatively impact the RTO and business requirement. Generally, backup device can be either one or combination of below devices:</span></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<ul>
<li><span style="text-indent: -18pt;"><span style="font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span><span style="text-indent: -18pt;">Disk Files – (SAN, NAS, etc.)</span></li>
<li><span style="text-indent: -18pt;"><span style="font-size: 7pt; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal;"> </span></span><span style="text-indent: -18pt;">Cloud Storage (Azure Blob Storage, Amazon Cloud Storage, etc.)</span></li>
</ul>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Best Practice:<o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Make sure, the backup device is easily accessible and secured. There are instances, SAN administrator mistakenly deleted the backup files while freeing disk spaces and other instances, where DBA does not have access to backup storage device and the person who is having access is on vacation. If you are in Enterprise setup, usually different things are managed by multiple persons, if you are in small or start-up setup, you might have access to everything. Plan your backup and recovery strategy accordingly, where to write the backup and how you can access the backup, when there is a need.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Also, there are compliance policy, where it demands the backup must be secured, might be using encryption algorithms.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Archival and Retention Policy:<o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Archival and Retention policy should be defined considering the legal requirements and compliance requirement. Depends upon your region compliance requirements may impact how you store and how long you retain the backup.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Restore:<o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b><br />
</b></div>
<div class="MsoNormal" style="text-align: justify;">
<b>Backup Availability:</b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
There are scenarios, where backup file got corrupted or mistakenly deleted from storage, if you have only one backup copy, it will lead to loss of data. Backup plan should consider this and implement, more than one copy of backup in an accessible location.</div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b> <b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Best Practice:</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Generally, it is good practice to have more than 2 copies of backup. Keep 1 one copy in the local machine or attached drive, other copy in SAN/NAS storage and another copy in remote location, such as cloud or other geo.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Keeping a copy in local help in reduce the time needed to transfer the file to local machine and it will help in achieving RTO.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<!-- Google Large Ad Starts --><br />
<script async="" src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<ins class="adsbygoogle" data-ad-client="ca-pub-9474984742004767" data-ad-format="fluid" data-ad-layout="in-article" data-ad-slot="6514005873" style="display: block; text-align: center;"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google Large Ad Ends --><br />
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Restoration Access:<o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">In a large enterprise setup, everyone may not have access to restore or recover data. These permissions include, access to backup storage, access to restoration and security access, if encryptions are in place. It is advisable to capture all these permissions in Backup and Recovery strategy and share it with business and technical stakeholders. <o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">In case of emergency, everyone knows their role and understand the sequence, when to execute their job. It will remove the confusions and keep the team in calm environment.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Documentation:<o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">I have interviewed a lot of people for the database administration role and I realized, the documentation is least important task in their day to day job. For a DBA, most of the tasks are repetitive and we are using a lot of scripts and automations. Even something developed or written by us few days back will be very difficult to understand later point of time. When you are in emergency, searching on Google or using others scripts will not help you and it seems it is simple to recover the data but it is not. It is mandatory to document each and every steps and business requirements in clear manner and communicate with all stakeholders.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Also, it is important to agree both the business and technical stakeholders on how quickly data can be retrieved and how much data can be lost and get sign-off on the documentation.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Testing:<o:p></o:p></span></b></div>
<div class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 3.0pt;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="background: white; line-height: normal; margin-bottom: 3.0pt;">
As John Ruskin said, <span style="color: #222222; mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: EN-IN;"><b>“Quality is never an accident; it is always the result of intelligent effort.”</b> It does not matter, how perfectly you designed your backup and recovery strategy, unless you tested many number of times and documented all the challenges and resolution, it is going to be surprise at the time of emergency.<o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">As a best practice, restore the backup on testing and staging frequently in a defined interval and restore on the production machine in longer intervals. There are instances, a backup copy may work in one machine with same database engine and configuration, may not work in other machine with same engine and configuration. Make a habit of test, test and test frequently in a defined period.</span></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br /></span></b>
<b style="mso-bidi-font-weight: normal;"><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">Summary:<o:p></o:p></span></b></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">A well-designed Backup and Recovery strategy will ensure the data availability and minimize the data lose. A documented strategy will act as guiding light in terms of emergency. It will remove the confusions and dependency on individuals. A good tested and orchestrated strategy ensures the business continuity and give guarantee to your sleep </span><span style="font-family: "wingdings"; mso-ascii-font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-char-type: symbol; mso-hansi-font-family: Calibri; mso-hansi-theme-font: minor-latin; mso-symbol-font-family: Wingdings;"><span style="mso-char-type: symbol; mso-symbol-font-family: Wingdings;">J</span></span><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">. <o:p></o:p></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;"><br />
</span></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin;">I hope this helps in planning your strategy, please share your thoughts on this article and let me know, if I need to edit or add content on this article. Thanks you for your time. <b style="mso-bidi-font-weight: normal;">Be proactive rather than being reactive!!!</b><o:p></o:p></span></div>
</div>
</div>
Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com7tag:blogger.com,1999:blog-2027709621250502166.post-87191500963441075932018-10-15T22:08:00.000+05:302018-10-22T13:34:00.696+05:30[Solved] MySQL User Operation - ERROR 1396 (HY000): Operation CREATE / DROP USER failed for 'user'@'host'<div dir="ltr" style="text-align: left;" trbidi="on"><div style="text-align: justify;"><span id="docs-internal-guid-b260dbd6-7fff-72fe-5a8b-3feb27068dd7"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Error Message: </span></span></div><div style="text-align: justify;"><span id="docs-internal-guid-7ee67408-7fff-97e2-301a-9b6bb2158bff"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">ERROR 1396 (HY000): Operation CREATE USER failed for 'admin'@'%'</span></span></div><div style="text-align: justify;"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;"><br />
</span></div><div style="text-align: justify;"><span id="docs-internal-guid-8628eed0-7fff-b2c3-782f-41be038a3a1f"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Generic Error Message:</span></span></div><div style="text-align: justify;"><span style="background-color: white; font-family: "calibri"; white-space: pre-wrap;">Operation %s failed for %s</span></div><div style="text-align: justify;"><span style="background-color: white; color: #555555; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;"><br />
</span></div><div style="text-align: justify;"><span id="docs-internal-guid-ef81d182-7fff-4b9e-0900-75b6c25ea12e"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Error Scenario:</span></span></div><span style="color: #666666; font-family: "calibri"; text-align: justify; white-space: pre-wrap;">Operation CREATE USER failed</span><br />
<span style="color: #666666; font-family: "calibri"; text-align: justify; white-space: pre-wrap;">Operation DROP USER failed</span><br />
<div style="text-align: justify;"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><br />
</span></div><div style="text-align: justify;"><span id="docs-internal-guid-e2fac9f8-7fff-fea3-711a-1eabd00989c6"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Reason:</span></span></div><div style="text-align: justify;"><span id="docs-internal-guid-dbf319d7-7fff-ded6-0502-e3b49347464c"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">The reason for this error is, you are trying to do some user operation but the user does not exist on the MySQL system. Also, for drop user, the user details are stored somewhere in the system, even though, you have already dropped the user from MySQL server.</span></span></div><div style="text-align: justify;"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;"><br />
</span></div><div style="text-align: justify;"><span id="docs-internal-guid-1f8ffd9f-7fff-0dfc-e6e6-1ddb7710391f"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Resolution:</span></span></div><div style="text-align: justify;"><span style="color: #666666; font-family: "calibri"; white-space: pre-wrap;">Revoke all access granted to the user, drop the user, and run FLUSH PRIVILEGE command to remove the caches. Now create/drop/alter the user, it will work.</span></div><div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;"><span style="background-color: transparent; color: #666666; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br />
</span></div><blockquote class="tr_bq" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #666666; font-family: "cousine"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">REVOKE ALL ON *.* FROM 'user'@'host';</span></blockquote><blockquote class="tr_bq" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: transparent; color: #666666; font-family: "cousine"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">DROP USER 'user'@'host';</span></blockquote><blockquote class="tr_bq" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="color: #666666; font-family: "cousine"; font-size: 11pt; white-space: pre-wrap;">FLUSH PRIVILEGES;</span></blockquote><div style="text-align: justify;"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><br />
</span></div><div style="text-align: justify;"><span id="docs-internal-guid-369eb278-7fff-75bc-837d-0ce3fd4a16f3"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Grant Tables:</span></span></div><span style="color: #666666; font-family: "calibri"; text-align: justify; white-space: pre-wrap;">The following tables will help you in identifying the user related informations (as of MySQL 5.7):</span><br />
<div dir="ltr" style="line-height: 1.2; margin-bottom: 11pt; margin-top: 0pt; text-align: justify;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: white; color: black; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br />
</span> <span style="background-color: white; color: black; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">mysql.user</span><span style="background-color: white; color: #555555; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: User accounts, global privileges, and other non-privilege columns</span></div><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: white; color: black; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">mysql.db</span><span style="background-color: white; color: #555555; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: Database-level privileges</span></div><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: white; color: black; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">mysql.tables_priv</span><span style="background-color: white; color: #555555; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: Table-level privileges</span></div><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: white; color: black; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">mysql.columns_priv</span><span style="background-color: white; color: #555555; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: Column-level privileges</span></div><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><span style="background-color: white; color: black; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">mysql.procs_priv</span><span style="background-color: white; color: #555555; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: Stored procedure and function privileges</span><br />
<span style="background-color: white; color: #555555; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><span id="docs-internal-guid-379b6191-7fff-45b0-b91d-8b486a875f4d"><span style="color: black; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">mysql.proxies_priv</span><span style="font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">: Proxy-user privilege</span></span></span></div><br />
</div><!-- Amazon Ads - Start --><br />
<iframe style="width:120px;height:240px;" marginwidth="0" marginheight="0" scrolling="no" frameborder="0" src="//ws-in.amazon-adsystem.com/widgets/q?ServiceVersion=20070822&OneJS=1&Operation=GetAdHtml&MarketPlace=IN&source=ac&ref=tf_til&ad_type=product_link&tracking_id=27049-21&marketplace=amazon®ion=IN&placement=1449314287&asins=1449314287&linkId=ed2759896705d5e94e7fb02800f8eada&show_border=true&link_opens_in_new_window=true&price_color=333333&title_color=0066c0&bg_color=ffffff"><br />
</iframe><br />
<!-- Amazon Ads - End --><br />
<div dir="ltr" style="line-height: 1.2; margin-bottom: 11pt; margin-top: 0pt; text-align: justify;"><span style="color: #666666; font-family: "calibri"; font-weight: 700; white-space: pre-wrap;">Related MySQL Bug Reports:</span></div><div dir="ltr" style="line-height: 1.2; margin-bottom: 11pt; margin-top: 0pt; text-align: justify;"><span id="docs-internal-guid-797e3318-7fff-de6b-90b1-56f7c51c8183"><a href="https://bugs.mysql.com/bug.php?id=28331" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">https://bugs.mysql.com/bug.php?id=28331</span></a></span></div><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;"><a href="https://bugs.mysql.com/bug.php?id=86523" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">https://bugs.mysql.com/bug.php?id=86523</span></a></div><div><span style="background-color: white; color: #555555; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span id="docs-internal-guid-afa84dc7-7fff-dc9d-0d53-51e66e67935f"></span></span><br />
<div style="text-align: justify;"><span style="background-color: white; color: #555555; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span id="docs-internal-guid-afa84dc7-7fff-dc9d-0d53-51e66e67935f"><span style="background-color: transparent; color: #666666;">I hope this post will help you, if you faced this error on some other scenarios or if you know, some other workaround / solution for this error, please add on the comment section. It will be helpful for other readers.</span></span></span></div><span style="background-color: white; color: #555555; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span id="docs-internal-guid-afa84dc7-7fff-dc9d-0d53-51e66e67935f"> </span></span></div><div style="text-align: justify;"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><br />
</span></div><div style="text-align: justify;"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><br />
</span></div><div style="text-align: justify;"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><br />
</span></div><div style="text-align: justify;"><span style="color: #666666; font-family: "calibri"; font-size: 11pt; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><br />
</span></div></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com5tag:blogger.com,1999:blog-2027709621250502166.post-9067106454027197392018-10-09T17:15:00.002+05:302018-10-22T13:41:00.391+05:30MySQL Replication Notes<div dir="ltr" style="text-align: left;" trbidi="on">
<span id="docs-internal-guid-7d1b59a6-7fff-6a76-d9d1-b23283b916c8"></span><br />
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-7d1b59a6-7fff-6a76-d9d1-b23283b916c8"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">The MySQL Replication was my first project as a Database Administrator (DBA) and I have been working with Replication technologies for last few years and I am indebted to contribute my little part for development of this technology. MySQL supports different replication topologies, having better understanding of basic concepts will help you in building and managing various and complex topologies.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-7d1b59a6-7fff-6a76-d9d1-b23283b916c8"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">I am writing here, some of the key points to taken care when you are building MySQL replication. I consider this post as a starting point for building a high performance and consistent MySQL servers. Let me start with below key points </span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span id="docs-internal-guid-7d1b59a6-7fff-6a76-d9d1-b23283b916c8"><span id="docs-internal-guid-1cef1712-7fff-bbd7-ba33-b672b8a9046b"><span style="background-color: transparent; color: #666666; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">
</span></span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span id="docs-internal-guid-7d1b59a6-7fff-6a76-d9d1-b23283b916c8"><span id="docs-internal-guid-1cef1712-7fff-bbd7-ba33-b672b8a9046b"><span style="background-color: transparent; color: #666666; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Hardware.</span></span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span id="docs-internal-guid-7d1b59a6-7fff-6a76-d9d1-b23283b916c8"><span id="docs-internal-guid-1cef1712-7fff-bbd7-ba33-b672b8a9046b"><span style="background-color: transparent; color: #666666; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">MySQL Server Version</span></span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span id="docs-internal-guid-7d1b59a6-7fff-6a76-d9d1-b23283b916c8"><span id="docs-internal-guid-1cef1712-7fff-bbd7-ba33-b672b8a9046b"><span style="background-color: transparent; color: #666666; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">MySQL Server Configuration</span></span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span id="docs-internal-guid-7d1b59a6-7fff-6a76-d9d1-b23283b916c8"><span id="docs-internal-guid-1cef1712-7fff-bbd7-ba33-b672b8a9046b"><span style="background-color: transparent; color: #666666; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Primary Key</span></span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span id="docs-internal-guid-7d1b59a6-7fff-6a76-d9d1-b23283b916c8"><span id="docs-internal-guid-1cef1712-7fff-bbd7-ba33-b672b8a9046b"><span id="docs-internal-guid-636cf7af-7fff-0b4f-b4de-95491f3573f5"></span></span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span id="docs-internal-guid-7d1b59a6-7fff-6a76-d9d1-b23283b916c8"><span id="docs-internal-guid-1cef1712-7fff-bbd7-ba33-b672b8a9046b"><span style="background-color: transparent; color: #666666; font-family: "calibri"; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Storage Engine</span></span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-7d1b59a6-7fff-6a76-d9d1-b23283b916c8"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">I will update this post with relevant points, whenever I get time. I am trying to provide generic concepts and it will be applicable to all version of MySQL, however, some of the concepts are new and applicable to latest versions (>5.0).</span></span></div>
<br />
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<span id="docs-internal-guid-1a04d252-7fff-a315-32be-c140f69379ac"><span style="color: #434343; font-family: "roboto slab"; font-size: 14pt; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Hardware:</span></span></div>
<div>
<span style="color: #666666; font-family: "calibri"; font-size: 11pt; text-align: justify; white-space: pre-wrap;">
</span></div>
<div>
<span style="color: #666666; font-family: "calibri"; text-align: justify; white-space: pre-wrap;">Resourcing of the slave must be on par (or better than) for any Master to keep up with the Master. The slave resource includes the following things:</span></div>
<div>
<br /></div>
<div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span id="docs-internal-guid-cdac7698-7fff-1112-c8de-f31d86115aaf"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">Disk IO</span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span id="docs-internal-guid-cdac7698-7fff-1112-c8de-f31d86115aaf"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">Computation (vCPU)</span></span></div>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span id="docs-internal-guid-cdac7698-7fff-1112-c8de-f31d86115aaf"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">InnoDB Buffer Pool (RAM)</span></span></div>
<div>
<span id="docs-internal-guid-cdac7698-7fff-1112-c8de-f31d86115aaf"><span style="color: #666666; font-family: "calibri"; text-align: justify; white-space: pre-wrap;">
</span></span></div>
<div>
<span id="docs-internal-guid-cdac7698-7fff-1112-c8de-f31d86115aaf"><span style="color: #666666; font-family: "calibri"; text-align: justify; white-space: pre-wrap;">MySQL 5.7 supports Multi Threaded Replication, but are limited to one thread per database. In case of heavy writes (multiple threads) on Master databases, there is a chance that, Slave will be lag behind the Master, since only one thread is applying BINLOG to the Slave per database and its writes are all serialised.</span></span></div>
</div>
<div style="text-align: left;">
<h1 dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 16pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #434343; font-family: "roboto slab"; font-size: 14pt; vertical-align: baseline; white-space: pre-wrap;">MySQL Version:</span></span></h1>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">It is highly recommended to have Master and Slave servers should run on same version. Different version of MySQL on slave can affect the SQL execution timings.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">For example, MySQL 8.0 is comparatively much faster than 5.5. Also, it is worth to consider the features addition, deletion and modifications.</span></span></div>
<h1 dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 16pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #434343; font-family: "roboto slab"; font-size: 14pt; vertical-align: baseline; white-space: pre-wrap;">MySQL Server Configuration:</span></span></h1>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">The MySQL server configuration should be identical, we may have identical hardware resources and same MySQL version, but if MySQL is not configured to utilize the available resources in similar method, there will be changes in execution plan.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">For example, InnoDB buffer pool size should be configured on MySQL server to utilize the memory. Even if we have a identical hardwares, buffer pool must be configured at the MySQL instance.</span></span></div>
<h1 dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 16pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #434343; font-family: "roboto slab"; font-size: 14pt; vertical-align: baseline; white-space: pre-wrap;">Primary Key:</span></span></h1>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">The primary key plays an important role in Row-Based-Replication (when binlog_format is either ROW or MIXED). Most often, slave lagging behind master while applying RBR event is due to the lack of primary key on the table involved.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">When no primary key is defined, for each affected row on master, the entire row image has to be compared on a row-by-row basis against the matching table’s data on the slave.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">This can be explained by how a transaction is performed on master and slave based on the availability of primary key:</span></span></div>
<div style="text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><br />
</span></div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none; text-align: justify;"><colgroup><col width="79"></col><col width="255"></col><col width="290"></col></colgroup><tbody>
<tr style="height: 0pt;"><td style="border-bottom: solid #000000 1pt; border-left: solid #000000 1pt; border-right: solid #000000 1pt; border-top: solid #000000 1pt; padding: 5pt 5pt 5pt 5pt; vertical-align: top;"><br /></td><td style="background-color: #efefef; border-bottom: solid #000000 1pt; border-left: solid #000000 1pt; border-right: solid #000000 1pt; border-top: solid #000000 1pt; padding: 5pt 5pt 5pt 5pt; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">With Primary Key</span></div>
</td><td style="background-color: #efefef; border-bottom: solid #000000 1pt; border-left: solid #000000 1pt; border-right: solid #000000 1pt; border-top: solid #000000 1pt; padding: 5pt 5pt 5pt 5pt; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="background-color: transparent; color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">Without Primary Key</span></div>
</td></tr>
<tr style="height: 0pt;"><td style="background-color: #d9d9d9; border-bottom: solid #000000 1pt; border-left: solid #000000 1pt; border-right: solid #000000 1pt; border-top: solid #000000 1pt; padding: 5pt 5pt 5pt 5pt; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">On Master</span></div>
</td><td style="border-bottom: solid #000000 1pt; border-left: solid #000000 1pt; border-right: solid #000000 1pt; border-top: solid #000000 1pt; padding: 5pt 5pt 5pt 5pt; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">Uniquely identifies the row</span></div>
</td><td style="border-bottom: solid #000000 1pt; border-left: solid #000000 1pt; border-right: solid #000000 1pt; border-top: solid #000000 1pt; padding: 5pt 5pt 5pt 5pt; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">Make use of any available key or performs a full table scan</span></div>
</td></tr>
<tr style="height: 0pt;"><td style="background-color: #d9d9d9; border-bottom: solid #000000 1pt; border-left: solid #000000 1pt; border-right: solid #000000 1pt; border-top: solid #000000 1pt; padding: 5pt 5pt 5pt 5pt; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="background-color: transparent; color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">On Slave</span></div>
</td><td style="border-bottom: solid #000000 1pt; border-left: solid #000000 1pt; border-right: solid #000000 1pt; border-top: solid #000000 1pt; padding: 5pt 5pt 5pt 5pt; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">Uniquely identifies each rows & changes can be quickly applied to appropriate row images on the slave.</span></div>
</td><td style="border-bottom: solid #000000 1pt; border-left: solid #000000 1pt; border-right: solid #000000 1pt; border-top: solid #000000 1pt; padding: 5pt 5pt 5pt 5pt; vertical-align: top;"><div dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: justify;">
<span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">Entire row image is compared on a row-by-row basis against the matching table’s data on slave.</span></div>
</td></tr>
</tbody></table>
</div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">Row-by-row scan can be very expensive and time consuming and cause slave to lag behind master.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">When there is no primary key defined on a table, InnoDB internally generates a hidden clustered index named GEN_CLUST_INDEX containing row ID values. MySQL replication cannot use this hidden primary key for sort operations, because this hidden row IDs are unique to each MySQL instance and are not consistent between a master and a slave.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">The best solution is to ensure all tables have a primary key. When there is no unique not null key available on table, at least create an auto-incrementing integer column (surrogate key) as primary key.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">If immediately, it is not possible to create a primary key on all such tables, there is a workaround to overcome this for short period of time by changing slave rows search algorithm. This is not the scope of this post, I will write future post on this topic.</span></span></div>
<h1 dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 16pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #434343; font-family: "roboto slab"; font-size: 14pt; vertical-align: baseline; white-space: pre-wrap;">Mixing of Storage Engines:</span></span></h1>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">MySQL Replication supports different storage engines on master and slave servers. But, there are few important configuration to be taken care when mixing of storage engines.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">It should be noted that, InnoDB is a transactional storage engine and MyISAM is a non-transactional. </span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">On Rollback:</span><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;"> If binlog_format is STATEMENT and when a transaction updates, InnoDB and MyISAM tables and then performs ROLLBACK, only InnoDB tables data is removed and when this statement is written to binlog it will be send to slave, on slave where both the tables are MyISAM will not perform the ROLLBACK, since it does not supports transaction. It will leave the table inconsistent with master.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Auto-Increment column: </span><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">This should be noted that, the way auto-increment is implemented on MyISAM and InnoDB different, MyISAM will lock a entire table to generate auto-increment and the auto-increment is part of a composite key, insert operation on MyISAM table marked as unsafe. Refer this page for better understanding </span><a href="https://dev.mysql.com/doc/refman/8.0/en/replication-features-auto-increment.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">https://dev.mysql.com/doc/refman/8.0/en/replication-features-auto-increment.html</span></a></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Referential Integrity Constraints: </span><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">InnoDB supports foreign keys and MyISAM does not. Cascading updates and deletes operations on InnoDB tables on master will replicate to slave, only if the tables are InnoDB on both master and slave. This is true for both STATEMENT and ROW based replications. Refer this page for explanation: </span><a href="https://dev.mysql.com/doc/refman/5.7/en/innodb-and-mysql-replication.html" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">https://dev.mysql.com/doc/refman/5.7/en/innodb-and-mysql-replication.html</span></a></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Locking: </span><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">InnoDB performs row-level locking and MyISAM performs table-level locking and all transaction on the slave are executed in a serialized manner, this will negatively impact the slave performance and end up in slave lagging behind the master.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Logging: </span><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">MyISAM is a non-transactional storage engine and transactions are logged into binary log by client thread, immediately after execution, but before the locks are released. </span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">If the query is part of the transaction and if there is a InnoDB table involved on same transaction and it is executed before the MyISAM query, then it will not written to binlog immediately after execution, it will wait for either commit or rollback. This is done to ensure, order of execution is same in slave as in the master.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">Transaction on InnoDB tables will be written to the binary log, only when the transaction is committed.</span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">It is highly advisable to use transactional storage engine on MySQL Replication. Mixing of storage engine may leads to inconsistency and performance issues between master and slave server. Though MySQL does not produce any warnings, it should be noted and taken care from our end. </span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">Also, the introduction of MySQL 8.0 (from 5.6) with default storage engine as InnoDB and deprecating older ISAM feature indicates the future of MySQL database, it is going to be completely transactional and it is recommended to have InnoDB storage engine. </span></span></div>
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">There is a discussion online, about the removal of other storage engines and development on InnoDB engine by Oracle, though it is not scope of this article, as a Database Administrator, I prefer having different storage engine for different use cases and it has been unique feature of MySQL.</span></span><br />
<span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;">
</span> <span style="color: #666666; font-family: "calibri"; vertical-align: baseline; white-space: pre-wrap;"><b>Reference:</b></span></div>
<!-- Amazon Ads - Start --><br />
<iframe frameborder="0" marginheight="0" marginwidth="0" scrolling="no" src="//ws-in.amazon-adsystem.com/widgets/q?ServiceVersion=20070822&OneJS=1&Operation=GetAdHtml&MarketPlace=IN&source=ac&ref=tf_til&ad_type=product_link&tracking_id=27049-21&marketplace=amazon&region=IN&placement=1449314287&asins=1449314287&linkId=ed2759896705d5e94e7fb02800f8eada&show_border=true&link_opens_in_new_window=true&price_color=333333&title_color=0066c0&bg_color=ffffff" style="height: 240px; width: 120px;"><br />
</iframe><br />
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;">
<span id="docs-internal-guid-ea79a181-7fff-f2b8-ffe6-bb878dbbe440"><span style="color: #666666; font-family: "calibri"; white-space: pre-wrap;">I hope this post is useful, please share your thoughts / feedbacks on comment section.</span></span></div>
</div>
</div>
Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com4tag:blogger.com,1999:blog-2027709621250502166.post-63231802404874225362018-09-18T21:44:00.001+05:302018-09-19T15:43:03.655+05:30[Solved] How to install MySQL Server on CentOS 7?<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span style="color: black; mso-themecolor: text1;">Recently, when I am working on setting
up MySQL Enterprise Server, I found, there is too much information available
over internet and it was very difficult for a newbie to get what is needed for
direct implementation. So, I decided to write a quick reference guide for
setting up the server, covering end to end, starting from planning to
production to maintenance. This is a first post in that direction, in this
post, we will discuss about installing MySQL Enterprise Server on CentOS 7
machine. Note that, the steps are same for both the Enterprise and Community
editions, only binary files are different, and downloaded from different
repositories.<o:p></o:p></span></div>
<div style="text-align: justify;">
<br /></div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span style="color: black; mso-themecolor: text1;">If you are
looking for installing MySQL on Windows operating system, please visit this
page </span><span class="MsoHyperlink"><span style="color: #0070c0;"><a href="https://www.rathishkumar.in/2016/01/how-to-install-mysql-server-on-windows.html"><span style="color: #0070c0;">https://www.rathishkumar.in/2016/01/how-to-install-mysql-server-on-windows.html</span></a></span></span><span class="MsoHyperlink"><span style="color: black; mso-themecolor: text1;">.</span></span><span style="color: black; mso-themecolor: text1;"> I am assuming, hardware and the
Operating System is installed and configured as per the requirement. Let us
begin with the installation.</span><br />
<span style="color: black; mso-themecolor: text1;"><br /></span></div>
</div>
<div style="text-align: justify;">
<span style="color: black; font-size: small;"><b>Removing
MariaDB:</b></span></div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span style="color: black; mso-themecolor: text1;"><br /></span>
<span style="color: black; mso-themecolor: text1;">The CentOS
comes with MariaDB as a default database, if you try to install, MySQL on top
of it, you will encounter an error message stating the MySQL library files
conflict with MariaDB library files. Remove the MariaDB to avoid errors and to
have a clean installation.<span style="mso-spacerun: yes;"> </span>Use below
statements to remove MariaDB completely:</span><br />
<span style="border: 1pt none; color: #0077aa; font-family: "consolas"; padding: 0cm;"><br /></span>
<span style="border: 1pt none; color: #0077aa; font-family: "consolas"; padding: 0cm;">sudo</span><span style="font-family: "consolas";">
yum remove MariaDB-server</span></div>
</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<span style="border: none 1.0pt; color: #0077aa; font-family: "consolas"; padding: 0cm;">sudo</span><span style="font-family: "consolas";">
yum remove MariaDB-client </span><span style="font-family: "consolas";">(This
can be done in single step)<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<span style="border: none 1.0pt; color: #0077aa; font-family: "consolas"; padding: 0cm;">sudo</span><span style="font-family: "consolas";">
rm –rf /var/lib/mysql</span></div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<span style="border: none 1.0pt; color: #0077aa; font-family: "consolas"; padding: 0cm;">sudo</span><span style="font-family: "consolas";">
rm /etc/my.cnf<o:p></o:p></span></div>
</div>
</div>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
(Run with sudo, if you are not logged in as
Super Admin).</div>
<div style="text-align: justify;">
<br />
<div class="MsoNormal">
<b>Downloading
RPM files:</b><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
</div>
<div style="text-align: justify;">
<div class="MsoNormal">
MySQL installation files (On CentOS 7 – rpm packages) can be downloaded from MySQL yum repository.</div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span style="color: black; mso-themecolor: text1;"><br /></span>
<span style="color: black; mso-themecolor: text1;">For MySQL
Community Edition – there is clear and step-by-step guide available at the
MySQL website - </span><span class="MsoHyperlink"><span style="color: #0070c0;"><a href="https://dev.mysql.com/doc/mysql-yum-repo-quick-guide/en/"><span style="color: #0070c0;">https://dev.mysql.com/doc/mysql-yum-repo-quick-guide/en/</span></a></span></span><span style="color: black; mso-themecolor: text1;">. The only step missing is downloading
MySQL yum repository to your local machine. (This might looks very simple step,
but most of the newbies, it is very helpful). </span><br />
<span style="color: black; mso-themecolor: text1;"><br /></span></div>
</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
<span style="font-family: "consolas";">wget
<span style="color: black; text-decoration-line: none;"><a href="https://dev.mysql.com/get/mysql80-community-release-el7-1.noarch.rpm">https://dev.mysql.com/get/mysql80-community-release-el7-1.noarch.rpm</a></span><o:p></o:p></span></div>
</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
<span style="font-family: "consolas";"><br /></span></div>
</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
For MySQL Enterprise Edition – the
binary files can be downloaded from Oracle Software Delivery Cloud (<span class="MsoHyperlink"><span style="color: #0070c0;"><a href="http://edelivery.oracle.com/"><span style="color: #0070c0;">http://edelivery.oracle.com</span></a></span></span>) for latest version or previous
versions visit My Oracle Support (<span class="MsoHyperlink"><span style="color: #0070c0;"><a href="https://support.oracle.com/"><span style="color: #0070c0;">https://support.oracle.com/</span></a></span></span>).<o:p></o:p></div>
</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
As mentioned earlier, there is a clear
and step-by-step guide available at the MySQL website for Community Edition, I
will be continue with installing Enterprise Edition, steps are almost same.</div>
</div>
<h3 style="text-align: justify;">
Choosing the
RPM file:</h3>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal">
<div style="text-align: justify;">
For MySQL
Community Edition, all the RPM files will be included in the downloaded YUM
repository, but for Enterprise Editions, these files will be downloaded
separately. (For system administration purpose, all these files can be created
under a MySQL repository).<o:p></o:p></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
For newbies,
it may be confusing to understand, the different RPM files and its contents, I
am concentrating on only files required for stand-alone MySQL instances. If
there is requirement for embedded MySQL or if you working on developing plugins
for MySQL, can install other files. It is completely depends on your
requirement. The following tables, describe the required files and where to
install.<o:p></o:p></div>
</div>
<div style="text-align: justify;">
<br /></div>
<table border="0" cellpadding="0" cellspacing="0" class="MsoNormalTable" style="border-collapse: collapse; text-align: justify;">
<tbody>
<tr>
<td style="background: #CCCCCC; border: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm; text-align: center;">
RPM File<span style="font-family: "times new roman" , serif; font-size: 12pt;"><o:p></o:p></span></div>
</td>
<td style="background: #CCCCCC; border-left: none; border: solid black 1.0pt; mso-border-left-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm; text-align: center;">
Description<span style="font-family: "times new roman" , serif; font-size: 12pt;"><o:p></o:p></span></div>
</td>
<td style="background: #CCCCCC; border-left: none; border: solid black 1.0pt; mso-border-left-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm; text-align: center;">
Location<span style="font-family: "times new roman" , serif; font-size: 12pt;"><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid black 1.0pt; mso-border-top-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<span style="font-family: "consolas";">mysql-commercial-server-5.7.23-1.1.el7.x86_64.rpm</span><span style="font-family: "consolas"; font-size: 12pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; mso-border-left-alt: solid black 1.0pt; mso-border-top-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm; text-align: justify; text-justify: inter-ideograph;">
<span style="background: white;">MySQL Server and related utilities to
run and administer a MySQL server.</span><span style="font-family: "times new roman" , serif; font-size: 12pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; mso-border-left-alt: solid black 1.0pt; mso-border-top-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm; text-align: justify; text-justify: inter-ideograph;">
<span style="background: white;">On Server</span><span style="font-family: "times new roman" , serif; font-size: 12pt;"><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid black 1.0pt; mso-border-top-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<span style="font-family: "consolas";">mysql-commercial-client-5.7.23-1.1.el7.x86_64.rpm</span><span style="font-family: "consolas"; font-size: 12pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; mso-border-left-alt: solid black 1.0pt; mso-border-top-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<span style="background: white;">Standard MySQL clients and
administration tools.</span><span style="font-family: "times new roman" , serif; font-size: 12pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; mso-border-left-alt: solid black 1.0pt; mso-border-top-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<span style="background: white;">On Server & On Client</span><span style="font-family: "times new roman" , serif; font-size: 12pt;"><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid black 1.0pt; mso-border-top-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<span style="font-family: "consolas";">mysql-commercial-common-5.7.23-1.1.el7.x86_64.rpm</span><span style="font-family: "consolas"; font-size: 12pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; mso-border-left-alt: solid black 1.0pt; mso-border-top-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm; text-align: justify; text-justify: inter-ideograph;">
<span style="background: white;">Common files needed by MySQL client
library, MySQL database server, and MySQL embedded server.</span><span style="font-family: "times new roman" , serif; font-size: 12pt;"><o:p></o:p></span></div>
</td>
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; mso-border-left-alt: solid black 1.0pt; mso-border-top-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm; text-align: justify; text-justify: inter-ideograph;">
<span style="background: white;">On Server</span><span style="font-family: "times new roman" , serif; font-size: 12pt;"><o:p></o:p></span></div>
</td>
</tr>
<tr>
<td style="border-top: none; border: solid black 1.0pt; mso-border-top-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<span style="font-family: "consolas";">mysql-commercial-libs-5.7.23-1.1.el7.x86_64.rpm<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; mso-border-left-alt: solid black 1.0pt; mso-border-top-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm; text-align: justify; text-justify: inter-ideograph;">
<span style="background: white;">Shared libraries for MySQL Client
applications<o:p></o:p></span></div>
</td>
<td style="border-bottom: solid black 1.0pt; border-left: none; border-right: solid black 1.0pt; border-top: none; mso-border-left-alt: solid black 1.0pt; mso-border-top-alt: solid black 1.0pt; padding: 5.0pt 5.0pt 5.0pt 5.0pt;" valign="top"><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm; text-align: justify; text-justify: inter-ideograph;">
<span style="background: white;">On Server<o:p></o:p></span></div>
</td>
</tr>
</tbody></table>
<div style="text-align: justify;">
<br /></div>
<div style="margin-bottom: 0.0001pt; text-align: justify;">
<b>
Installing MySQL:</b></div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<o:p></o:p></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Install the MySQL binary files in the following
order, this is to avoid dependency errors, the following statements will
install MySQL on local machine:</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
<span style="color: black; mso-ascii-font-family: Calibri; mso-bidi-font-family: Calibri; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: "Times New Roman"; mso-hansi-font-family: Calibri; mso-themecolor: text1;"><br /></span></div>
</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
<span style="color: #0070c0; font-family: "consolas"; mso-bidi-font-family: Calibri; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: "Times New Roman";">sudo </span><span style="color: black; font-family: "consolas"; mso-bidi-font-family: Calibri; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: "Times New Roman"; mso-themecolor: text1;">yum
localinstall mysql-commercial-libs-5.7.23-1.1.el7.x86_64.rpm<o:p></o:p></span></div>
</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
<span style="color: #0070c0; font-family: "consolas"; mso-bidi-font-family: Calibri; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: "Times New Roman";">sudo </span><span style="color: black; font-family: "consolas"; mso-bidi-font-family: Calibri; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: "Times New Roman"; mso-themecolor: text1;">yum
localinstall mysql-commercial-client-5.7.23-1.1.el7.x86_64.rpm<o:p></o:p></span></div>
</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
<span style="color: #0070c0; font-family: "consolas"; mso-bidi-font-family: Calibri; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: "Times New Roman";">sudo </span><span style="color: black; font-family: "consolas"; mso-bidi-font-family: Calibri; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: "Times New Roman"; mso-themecolor: text1;">yum
localinstall mysql-commercial-common-5.7.23-1.1.el7.x86_64.rpm<o:p></o:p></span></div>
</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
<span style="color: #0070c0; font-family: "consolas"; mso-bidi-font-family: Calibri; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: "Times New Roman";">sudo </span><span style="color: black; font-family: "consolas"; mso-bidi-font-family: Calibri; mso-bidi-font-size: 11.0pt; mso-fareast-font-family: "Times New Roman"; mso-themecolor: text1;">yum
localinstall mysql-commercial-server-5.7.23-1.1.el7.x86_64.rpm<o:p></o:p></span></div>
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Starting the
MySQL Service:</b></div>
<div class="MsoNormal">
<o:p></o:p></div>
<div style="text-align: justify;">
<br /></div>
<div class="MsoNormal">
<div style="text-align: justify;">
On CentOS 7,
the mysql service can be started by following:<o:p></o:p></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
<span style="border: none 1.0pt; color: #0077aa; font-family: "consolas"; padding: 0cm;">sudo</span><span style="font-family: "consolas";">
systemctl start mysqld.service<o:p></o:p></span></div>
</div>
<div style="text-align: justify;">
<span style="border: 1pt none; color: #0077aa; font-family: "consolas"; padding: 0cm;">sudo</span><span style="font-family: "consolas";">
systemctl status mysqld.service</span></div>
<div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;">
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Login to MySQL Server for first time:</b></div>
</div>
<div>
<div style="text-align: justify;">
<span style="font-family: "calibri" , sans-serif; font-size: 11pt;"><br /></span></div>
</div>
<div>
<div class="MsoNormal">
<div style="text-align: justify;">
Once the
service is started, the superuser account ‘root’@’localhost’ created and
temporary password is stored at the error log file (default <span style="font-family: "consolas";">/var/log/mysqld.log</span>). The password can be retrieved by
using the following command:<o:p></o:p></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span style="border: none 1.0pt; color: #0077aa; font-family: "consolas"; padding: 0cm;">sudo</span><span style="border: 1pt none; font-family: "consolas"; padding: 0cm;"> grep </span><span style="border: none 1.0pt; color: #0077aa; font-family: "consolas"; padding: 0cm;">'temporary password'</span><span style="border: 1pt none; font-family: "consolas"; padding: 0cm;"> /var/log/mysqld</span><span style="border: none 1.0pt; color: #999999; font-family: "consolas"; padding: 0cm;">.</span><span style="border: 1pt none; font-family: "consolas"; padding: 0cm;">log</span><o:p></o:p></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span style="border: 1pt none; font-family: "consolas"; padding: 0cm;"><br /></span></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
As soon as logged in to MySQL with the temporary password,
need to reset the root password, until that, you cannot run any queries on
MySQL server. You can reset the root account password by running below command.<o:p></o:p></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #0077aa; font-family: "consolas"; padding: 0cm;">mysql –u root –h localhost
-p<o:p></o:p></span></span></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #0077aa; font-family: "consolas"; padding: 0cm;">alter user</span></span><span style="background: rgb(248 , 248 , 248); font-family: "consolas";"> </span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #669900; font-family: "consolas"; padding: 0cm;">'root'</span></span><span style="background: rgb(248 , 248 , 248); font-family: "consolas";">@</span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #669900; font-family: "consolas"; padding: 0cm;">'localhost'</span></span><span style="background: rgb(248 , 248 , 248); font-family: "consolas";"> </span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #0077aa; font-family: "consolas"; padding: 0cm;">identified by</span></span><span style="background: rgb(248 , 248 , 248); font-family: "consolas";"> </span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #669900; font-family: "consolas"; padding: 0cm;">'NewPassword'</span></span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #999999; font-family: "consolas"; padding: 0cm;">;</span></span></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #999999; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
You can verify the MySQL status and edition by running the
following commands, sample output provided below for MySQL 8.0 Community
Edition (GPL License) running on Windows machine.<o:p></o:p></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3e4fn44RCPuDSbple4WRJ_3PnaB53F5MgbJtPOmsR2PUIJsezpdCsoh_x2dqvTqDtarXSNLEybczzRa8Out0gGpZTn34iVVmtATG4smzojpOknC35nI8OGEPGFYLI4mr_PGv53v2rwBNt/s1600/MySQL+License+%2526+Status.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="657" data-original-width="985" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3e4fn44RCPuDSbple4WRJ_3PnaB53F5MgbJtPOmsR2PUIJsezpdCsoh_x2dqvTqDtarXSNLEybczzRa8Out0gGpZTn34iVVmtATG4smzojpOknC35nI8OGEPGFYLI4mr_PGv53v2rwBNt/s640/MySQL+License+%2526+Status.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">MySQL License Status</td></tr>
</tbody></table>
</div>
<div>
<div style="text-align: justify;">
<b>
Notes:</b></div>
<div style="text-align: justify;">
<b><br /></b></div>
<div class="MsoNormal">
<o:p></o:p></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<b>MySQL conflicts with
MariaDB:</b> in case if there is conflict with MariaDB, you will see the error
message as below:<o:p></o:p></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span class="kwd"><span style="background: #eff0f1; border: none 1.0pt; color: #101094; font-family: "consolas"; padding: 0cm;">file</span></span><span class="pln"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;"> </span></span><span class="pun"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;">/</span></span><span class="pln"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;">usr</span></span><span class="pun"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;">/</span></span><span class="pln"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;">share</span></span><span class="pun"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;">/</span></span><span class="pln"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;">mysql</span></span><span class="pun"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;">/</span></span><span class="pln"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;">xxx </span></span><span class="kwd"><span style="background: #eff0f1; border: none 1.0pt; color: #101094; font-family: "consolas"; padding: 0cm;">from</span></span><span class="pln"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;"> install </span></span><span class="kwd"><span style="background: #eff0f1; border: none 1.0pt; color: #101094; font-family: "consolas"; padding: 0cm;">of</span></span><span class="pln"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;">
MySQL-server-xxx conflicts </span></span><span class="kwd"><span style="background: #eff0f1; border: none 1.0pt; color: #101094; font-family: "consolas"; padding: 0cm;">with</span></span><span class="pln"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;"> </span></span><span class="kwd"><span style="background: #eff0f1; border: none 1.0pt; color: #101094; font-family: "consolas"; padding: 0cm;">file</span></span><span class="pln"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;"> </span></span><span class="kwd"><span style="background: #eff0f1; border: none 1.0pt; color: #101094; font-family: "consolas"; padding: 0cm;">from</span></span><span class="pln"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;"> package mariadb-libs-xxx</span></span><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;"><o:p></o:p></span></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<span class="pln"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
</div>
<div class="MsoNormal">
<span class="pln"><span style="background: #eff0f1; border: none 1.0pt; color: #303336; font-family: "consolas"; padding: 0cm;"></span></span></div>
<div class="MsoNormal">
<div style="text-align: justify;">
To resolve this error remove mariadb server and its related
files from CentOS server. Refer the section - Removing mariadb.<o:p></o:p></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<b>Can’t connect to
mysql server: </b>MySQL server is installed but unable to connect from client.<o:p></o:p></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
Check this page for possible causes and solutions: <a href="https://www.rathishkumar.in/2017/08/solved-cannot-connect-to-mysql-server.html"><span style="color: #0b5394;">https://www.rathishkumar.in/2017/08/solved-cannot-connect-to-mysql-server.html</span></a>
<o:p></o:p></div>
</div>
<div class="MsoNormal">
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
<br /></div>
</div>
<div class="MsoNormal">
<div style="text-align: justify;">
Please let me know, if you are facing any other errors on
comment section. I hope this post is helpful.</div>
</div>
</div>
Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com11tag:blogger.com,1999:blog-2027709621250502166.post-91866978517302847382018-06-15T15:52:00.001+05:302018-06-15T16:18:45.346+05:30[Solved] Unable to start SQL Server Service after applying service pack Error: Script level upgrade for database 'master' failed because upgrade step 'ISServer_upgrade.sql' encountered error 5069, state 1, severity 16<div dir="ltr" style="text-align: left;" trbidi="on"><div style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;"><span style="background-color: transparent; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="color: #38761d; font-family: inherit;">Problem:</span></span></div><div style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;"><span style="white-space: pre-wrap;"><span style="font-family: inherit;">SQL Server service not starting after service pack / cumulative update installation.</span></span></div><div style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;"><span style="color: #38761d; font-weight: 700; white-space: pre-wrap;">Error Message:</span></div><div style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;"><span style="white-space: pre-wrap;">Script level upgrade for database 'master' failed because upgrade step 'ISServer_upgrade.sql' encountered error 5069, state 1, severity 16. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the 'master' database, it will prevent the entire SQL Server instance from starting. Examine the previous error log entries for errors, take the appropriate corrective actions and restart the database so that the script upgrade steps run to completion.</span></div><div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;"><span id="docs-internal-guid-07ac8460-0259-d076-f729-5273bba473d9"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><span style="color: #38761d; font-family: inherit;">Root Cause:</span></span></span></div><div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt;"><span style="font-family: inherit; text-align: justify; vertical-align: baseline; white-space: pre-wrap;">From primary analysis, the code </span><span style="font-family: inherit; text-align: justify; vertical-align: baseline; white-space: pre-wrap;">ISServer_upgrade.sql </span><span style="font-family: inherit; text-align: justify; vertical-align: baseline; white-space: pre-wrap;">is part of SQL Server DLL and it is no where located on the operating system.</span></div><div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt;"><span style="font-family: inherit; text-align: justify; white-space: pre-wrap;">Upgrading this script failing for some reason, which can be identified by starting sql server with trace 3601 and check the error log to identify the exact code running on SQL Server while updating the server.</span></div><div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify; text-indent: 0px;"><span style="font-family: inherit; text-indent: 36pt; white-space: pre-wrap;"> </span><span style="font-family: inherit; font-weight: 700; text-indent: 36pt; white-space: pre-wrap;">NET START MSSQLServer /T902 /T3601</span></div><div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify; text-indent: 0px;"><span style="font-family: inherit; white-space: pre-wrap;">However, this DLL is part of the SQL Server, this will be fixed by updating the next service pack or cumulative update.</span></div><div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify; text-indent: 0px;"><div style="text-align: justify;"><span style="color: #38761d; font-family: inherit; font-weight: 700; text-align: left; white-space: pre-wrap;">Troubleshooting Steps:</span></div></div><div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify; text-indent: 0px;">Start the SQL Server Service with trace 902.</div><div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify; text-indent: 0px;"> <b>NET START MSSQLServer /T902 </b></div><!-- Google Ad-Starts --><br />
<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({
google_ad_client: "ca-pub-9474984742004767",
enable_page_level_ads: true
});
</script><br />
<!-- Google Ad-Ends --><br />
<div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify; text-indent: 0px;">Download the latest cumulative update on top of service pack.<br />
<br />
<span style="font-family: inherit; font-weight: 700; white-space: pre-wrap;"> For example, after installing Service Pack 2 on SQL Server 2016 RTM, if the SQL Server Service not started, install the latest cumulative update CU2 on top of Service Pack 2 while running SQL on T902.</span></div><div style="text-align: justify;"><b style="white-space: pre-wrap;"><span style="font-weight: normal;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><span style="font-family: inherit;"> </span></span></span></span></span></b></div><div style="text-align: justify;"><b style="white-space: pre-wrap;"><span style="font-weight: normal;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><span id="docs-internal-guid-55b45755-0261-0f68-aa9e-85819e9c2cfe"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><span style="font-family: inherit;">After successfully updating SQL Server with CU2 stop the 902 trace and restart the SQL Server Service from SQL Server Configuration Manager.</span></span></span></span></span></span></b></div><div style="text-align: justify;"><span style="font-family: inherit; font-weight: 700; white-space: pre-wrap;"> </span></div><div style="text-align: justify;"><span style="font-family: inherit; white-space: pre-wrap;"> </span><span style="font-family: inherit; font-weight: 700; white-space: pre-wrap;">NET STOP MSSQLServer</span><br />
<div style="text-indent: 0px;"><span style="font-family: inherit; font-weight: 700; text-indent: 36pt; white-space: pre-wrap;"> NET START MSSQLServer </span></div></div><div><div style="text-align: justify;"><span id="docs-internal-guid-285eb887-0262-8a2e-ce13-7a9251633299"><span style="font-family: inherit;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"> </span></span></span></div><div style="text-align: justify;"><span id="docs-internal-guid-285eb887-0262-8a2e-ce13-7a9251633299"><span style="font-family: inherit;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"><span id="docs-internal-guid-37c23f9a-0262-ee60-c5cd-22b774b8092f" style="font-weight: normal;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline;"><span style="color: #38761d;">Reference:</span></span></span></span></span></span></div><div style="text-align: justify;"><span style="font-family: inherit; white-space: pre-wrap;">SQL Server build version:</span></div></div><div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt;"><span style="font-family: inherit;"><span id="docs-internal-guid-ed30434f-0263-6b6c-4f1f-a0c0bdb1e59b"></span></span></div><div dir="ltr" style="line-height: 1.56; margin-bottom: 0pt; margin-top: 10pt; text-align: justify;"><a href="https://support.microsoft.com/en-in/help/321185/how-to-determine-the-version-edition-and-update-level-of-sql-server-an" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: inherit;">https://support.microsoft.com/en-in/help/321185/how-to-determine-the-version-edition-and-update-level-of-sql-server-an</span></span></a></div></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com0tag:blogger.com,1999:blog-2027709621250502166.post-74778134574527515432018-05-19T23:39:00.001+05:302018-05-20T00:12:41.159+05:30How to set up MySQL InnoDB Cluster? Part One<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div class="MsoNormal" style="text-align: justify;">
This post is about setting up MySQL InnoDB Cluster with 5
nodes on a sandbox deployment. Here, we focus on implementation part, the core
concepts will be explained in separate posts.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal">
<b>Prerequisites:</b></div>
<div class="MsoNormal">
</div>
<ul style="text-align: left;">
<li>MySQL Engine</li>
<li>MySQL Shell</li>
<li>MySQL Router</li>
</ul>
<div style="text-align: justify;">
Deploying MySQL InnoDB Cluster
involves the following steps:</div>
<ul style="text-align: left;">
<li>Deploying MySQL Engine (Sandbox
Instance)</li>
<li>Creating an InnoDB Cluster</li>
<li>Adding nodes to InnoDB Cluster</li>
<li>Configuring MySQL Router for High
Availability.</li>
<li>Testing High Availability.</li>
</ul>
<div>
<a name='more'></a><br /></div>
<b>Deploying MySQL
Engine:</b><br />
<div>
<br /></div>
<div>
<div style="text-align: justify;">
If the MySQL engines are already installed on all the nodes,
you can skip this step and directly move into creating an InnoDB Cluster part.</div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
I am deploying 5 Sandbox instances (which is in-built on
MySQL Shell application) on a same machine. On production system, there will be
separate nodes for each MySQL Engines. Let’s begin with the deployments:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<div class="MsoNormal">
To open MySQL Shell :
<span class="token"><span style="background: #F8F8F8; border: none windowtext 1.0pt; color: #a67f59; font-family: Consolas; mso-border-alt: none windowtext 0cm; padding: 0cm;">Start ->
cmd -> Type mysqlsh (OR) Start -> MySQL Shell</span></span> <o:p></o:p></div>
<br />
<div class="MsoNormal">
To change script mode :
<span class="token"><span style="background: #F8F8F8; border: none windowtext 1.0pt; color: #a67f59; font-family: Consolas; mso-border-alt: none windowtext 0cm; padding: 0cm;">\JS –
JavaScript Mode | \PY – Python Mode | \SQL – SQL Mode<o:p></o:p></span></span></div>
</div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="background: white; mso-background-themecolor: background1; text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL JS >
dba.deploySandboxInstance(port)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="background: white; mso-background-themecolor: background1; text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="background: white; mso-background-themecolor: background1; text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"></span></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">deploySandboxInstance()</span></span>
module will deploy new Sandbox Instance on the mentioned port, let’s deploy the
following 5 Sandbox instances:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">dba.deploySandboxInstance (3307)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">dba.deploySandboxInstance (3308)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">dba.deploySandboxInstance (3309)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">dba.deploySandboxInstance (3310)<o:p></o:p></span></span></div>
<div class="MsoNormal">
</div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">dba.deploySandboxInstance (3311)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"></span></span></div>
<div class="MsoNormal">
<b>Sample Output:<o:p></o:p></b></div>
<div class="MsoNormal">
<b><br /></b></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL JS > dba.deploySandboxInstance (3307)<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">A new MySQL sandbox instance will be created on
this host in<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">C:\Users\rathish.kumar\MySQL\mysql-sandboxes\3307<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Warning: Sandbox instances are only suitable
for deploying and running on your local machine for testing purposes and are
not accessible from external networks.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Please enter a MySQL root password for the new
instance: ***<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Deploying new MySQL instance...<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Instance localhost: 3307 successfully deployed
and started.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Use shell.connect('root@localhost:3307'); to
connect to the instance.<o:p></o:p></span></span></div>
<div class="MsoNormal">
</div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL JS ><o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal">
<b>To connect the
deployed sandbox instance:<o:p></o:p></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL JS > \connect user@host:port</span></span>
and enter the password when prompted. (OR)<o:p></o:p></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">
</span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL JS > shell.connect(‘user@host:port’)<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal">
<b>Sample Output:<o:p></o:p></b></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL localhost: 3307 ssl JS > \connect
root@localhost:3307<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Creating a session to 'root localhost: 3307’<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Enter password: ***<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Fetching schema names for auto completion...
Press ^C to stop.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Closing old connection...<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Your MySQL connection id is 16<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Server version: 8.0.11 MySQL Community Server -
GPL<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">No default schema selected; type \use
<schema> to set one.<o:p></o:p></schema></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL localhost: 3307 ssl JS > \ssl<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Switching to SQL mode... Commands end with;<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL localhost: 3307 ssl SQL > select
@@port;<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">| @@port |<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">| 3307
|<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">1 row in set (0.0006 sec)<o:p></o:p></span></span></div>
<div class="MsoNormal">
</div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL localhost: 3307 ssl SQL ><o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal">
<b>Creating InnoDB
Cluster:<o:p></o:p></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
To create an InnoDB cluster, connect to seed (primary)
server, which contains the original data by using above method and follow the
below steps:</div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">var cluster = dba.createCluster('ClusterName')</span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<b style="text-align: left;"><br /></b></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<b style="text-align: left;">Sample Output:</b></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span style="background-color: #f8f8f8; color: #a67f59; font-family: "consolas"; text-align: left;"><br /></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span style="background-color: #f8f8f8; color: #a67f59; font-family: "consolas"; text-align: left;">MySQL </span><span style="background-color: #f8f8f8; color: #a67f59; font-family: "consolas"; text-align: left;">localhost:3307 ssl</span><span style="color: #a67f59; font-family: "consolas"; text-align: left;"> </span><span style="background-color: #f8f8f8; color: #a67f59; font-family: "consolas"; text-align: left;">JS > var
cluster = dba.createCluster('DBCluster')</span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span style="background-color: #f8f8f8; color: #a67f59; font-family: "consolas"; text-align: left;">A new InnoDB cluster will be created on
instance 'root@localhost:3307'.</span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Validating instance at
localhost:3307...Instance detected as a sandbox.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Please note that sandbox instances are only
suitable for deploying test clusters for use within the same host.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">This instance reports its own address as
L-IS-RATHISH<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Instance configuration is suitable.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Creating InnoDB cluster 'DBCluster' on
'root@localhost:3307'...<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Adding Seed Instance...<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Cluster successfully created. Use
Cluster.addInstance() to add MySQL instances.<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">At least 3 instances are needed for the cluster
to be able to withstand up to one server failure.</span></span></div>
<div class="MsoNormal">
<b><br /></b></div>
<div class="MsoNormal">
<b>Adding nodes to
InnoDB Cluster:</b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
The secondary replication nodes will be added to cluster by
using the <span style="background-color: #f8f8f8; color: #dd4a68; font-family: "consolas"; text-align: justify;">addInstance()</span> method.</div>
<div class="MsoNormal">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql-js></span></span><span style="background: #f8f8f8; color: black; font-family: "consolas";"> cluster</span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #999999; font-family: "consolas"; padding: 0cm;">.</span></span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #dd4a68; font-family: "consolas"; padding: 0cm;">addInstance</span></span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #999999; font-family: "consolas"; padding: 0cm;">(</span></span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #669900; font-family: "consolas"; padding: 0cm;">'user@host:port'</span></span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #999999; font-family: "consolas"; padding: 0cm;">)</span></span><span style="font-size: 14.0pt; mso-bidi-font-size: 10.0pt;"><o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Let us add the nodes, one by one:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">cluster.addInstance('root@localhost:3308');<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">cluster.addInstance('root@localhost:3309');<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">cluster.addInstance('root@localhost:3310');<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">cluster.addInstance('root@localhost:3311');</span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<b style="text-align: left;"><br /></b></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<b style="text-align: left;">Sample Output:</b></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="text-align: left;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL<span style="mso-spacerun: yes;">
</span>localhost:3307 ssl<span style="mso-spacerun: yes;"> </span>JS >
cluster.addInstance('root@localhost:3311');<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">A new instance will be added to the InnoDB cluster.
Depending on the amount of data on the cluster this might take from a few
seconds to several hours.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Please provide the password for
'root@localhost:3311': ***<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Adding instance to the cluster ...<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Validating instance at localhost:3311...<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Instance detected as a sandbox.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Please note that sandbox instances are only
suitable for deploying test clusters for use within the same host.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">This instance reports its own address as
L-IS-RATHISH<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Instance configuration is suitable.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">The instance 'root@localhost:3311' was
successfully added to the cluster.</span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<b style="text-align: left;"><br /></b></div>
<div class="MsoNormal" style="text-align: justify;">
<b style="text-align: left;">Configuring MySQL
Router for High Availability:</b></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
MySQL Router routes client connections to servers in the
cluster and it provides separate ports for Read and Read/Write operations.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
MySQL Router takes its configuration from InnoDB Cluster’s
metadata and configure itself by using <span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">–-bootstrap </span></span>option.
It is recommended to install MySQL Router on a separate server or can be
installed on the application server.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
The MySQL Router command is given below, this should be run
on the server with Read/Write (R/W) role.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="text-align: justify;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">shell></span></span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #0077aa; font-family: "consolas"; padding: 0cm;"> mysqlrouter</span></span><span style="background: #f8f8f8; color: black; font-family: "consolas";"> </span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #990055; font-family: "consolas"; padding: 0cm;">--bootstrap</span></span><span style="background: #f8f8f8; color: black; font-family: "consolas";"> user@host</span><span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #999999; font-family: "consolas"; padding: 0cm;">:</span></span><span style="background: #f8f8f8; color: black; font-family: "consolas";">port</span><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
The server roles can be checked by using the status()
method. Let us check the status of our cluster:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL<span style="mso-spacerun: yes;"> </span>localhost:3307
ssl<span style="mso-spacerun: yes;"> </span>JS > cluster.status()<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">{<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"clusterName": "DBCluster",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"defaultReplicaSet": {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"name": "default",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"primary": "localhost:3307",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"ssl": "REQUIRED",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"status": "OK",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"statusText": "Cluster is ONLINE and can tolerate up to 2
failures.",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"topology": {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"localhost:3307": {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"address": "localhost:3307",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"mode": "R/W",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"readReplicas": {},<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span><span style="mso-spacerun: yes;"> </span>"role":
"HA",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"status": "ONLINE"<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>},<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"localhost:3308": {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"address": "localhost:3308",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"mode": "R/O",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"readReplicas": {},<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"role": "HA",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span><span style="mso-spacerun: yes;"> </span>"status":
"ONLINE"<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>},<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"localhost:3309": {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"address": "localhost:3309",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"mode": "R/O",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"readReplicas": {},<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"role": "HA",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"status": "ONLINE"<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>},<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"localhost:3310": {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"address": "localhost:3310",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"mode": "R/O",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"readReplicas": {},<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"role": "HA",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"status": "ONLINE"<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>},<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"localhost:3311": {<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"address": "localhost:3311",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"mode": "R/O",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"readReplicas": {},<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"role": "HA",<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"status": "ONLINE"<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>}<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;"> </span>}<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;"> </span>},<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;">
</span>"groupInformationSourceMember":
"mysql://root@localhost:3307"<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">}<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><span style="mso-spacerun: yes;"> </span>MySQL<span style="mso-spacerun: yes;">
</span>localhost:3307 ssl<span style="mso-spacerun: yes;"> </span>JS ><o:p></o:p></span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
The server root@localhost:3307 is currently assigned with
R/W role. Configure MySQL Router on this server:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">C:\Windows\system32>mysqlrouter --bootstrap
root@localhost:3307<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Please enter MySQL password for root:<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Reconfiguring system MySQL Router instance...<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">WARNING: router_id 1 not found in metadata<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL Router has now been configured for the
InnoDB cluster 'DBCluster'.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">The following connection information can be
used to connect to the cluster.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Classic MySQL protocol connections to cluster
'DBCluster':<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">- Read/Write Connections: localhost:6446<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">- Read/Only Connections: localhost:6447<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">X protocol connections to cluster 'DBCluster':<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">- Read/Write Connections: localhost:64460<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">- Read/Only Connections: localhost:64470<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Existing configurations backed up to
'C:/Program Files/MySQL/MySQL Router 8.0/mysqlrouter.conf.bak'<o:p></o:p></span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Connecting InnoDB
Cluster:<o:p></o:p></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
From MySQL Router configuration, we get the connection
information, by default, port 6446 used for Read /Write connections and Port
6447 used for Read/Only connections. MySQL Router allows to configure custom
port numbers for R/W and R/O client connections.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Let us connect to first connect to Read/Write port and then
connect to Read/Only port for testing.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Read/Write Instance:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">C:\Users\rathish.kumar>mysql -u root -h
localhost -P6446 -p<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Enter password: *<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Welcome to the MySQL monitor.<span style="mso-spacerun: yes;"> </span>Commands end with ; or \g.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Your MySQL connection id is 176<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Server version: 8.0.11 MySQL Community Server -
GPL<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Copyright (c) 2000, 2018, Oracle and/or its affiliates.
All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or
its affiliates. Other names may be trademarks of their respective owners.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Type 'help;' or '\h' for help. Type '\c' to
clear the current input statement.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql> select @@port;<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">| @@port |<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">|<span style="mso-spacerun: yes;"> </span>3307
|<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">1 row in set (0.00 sec)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql> create database ClustDB;<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Query OK, 1 row affected (0.09 sec)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql> use ClustDB;<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Database changed<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql> create table t1 (id int auto_increment
primary key);<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Query OK, 0 rows affected (0.18 sec)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql> insert into t1 (id) values(1);<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Query OK, 1 row affected (0.06 sec)<o:p></o:p></span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Read/Only Instance:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">C:\Users\rathish.kumar>mysql -u root -h
localhost -P6447 -p<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Enter password: *<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Welcome to the MySQL monitor.<span style="mso-spacerun: yes;"> </span>Commands end with ; or \g.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Your MySQL connection id is 47<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Server version: 8.0.11 MySQL Community Server -
GPL<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Copyright (c) 2000, 2018, Oracle and/or its affiliates.
All rights reserved. Oracle is a registered trademark of Oracle Corporation
and/or its affiliates. Other names may be trademarks of their respective owners.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Type 'help;' or '\h' for help. Type '\c' to clear
the current input statement.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql> select @@port;<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">| @@port |<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">|<span style="mso-spacerun: yes;"> </span>3308
|<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">1 row in set (0.00 sec)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql> select * from ClustDB.t1;<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+----+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">| id |<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+----+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">|<span style="mso-spacerun: yes;"> </span>1 |<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+----+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">1 row in set (0.00 sec)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql> insert into ClustDB.t1 (id) values
(2);<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">ERROR 1290 (HY000): The MySQL server is running
with the --super-read-only option so it cannot execute this statement<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql><o:p></o:p></span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Testing High Availability:<o:p></o:p></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
We have connected to R/W and R/O instances, and it is
working as expected. Now let’s test the High Availability by killing primary
seed node (3307) and Read/Only instance (3308).<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">dba.killSandboxInstance(3307)<o:p></o:p></span></span></div>
<div class="MsoNormal">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">dba.killSandboxInstance(3308)<o:p></o:p></span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Sample output:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">MySQL<span style="mso-spacerun: yes;">
</span>localhost:3307 ssl<span style="mso-spacerun: yes;"> </span>JS >
dba.killSandboxInstance(3307);<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">The MySQL sandbox instance on this host in<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">C:\Users\rathish.kumar\MySQL\mysql-sandboxes\3307
will be killed<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Killing MySQL instance...<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Instance localhost:3307 successfully killed.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="tab-stops: 492.0pt;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Now refresh run the query on the
existing Read/Write and Read/Only connections and check the port:<span style="mso-tab-count: 1;"> </span><o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="tab-stops: 492.0pt;">
<b style="mso-bidi-font-weight: normal;">Read/Only Instance: <o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql> select @@port;<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">ERROR 2006 (HY000): MySQL server has gone away<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">No connection. Trying to reconnect...<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Connection id:<span style="mso-spacerun: yes;"> </span>38<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Current database: *** NONE ***<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">| @@port |<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">|<span style="mso-spacerun: yes;"> </span>3310
|<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">1 row in set (1.30 sec)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql><o:p></o:p></span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
This error is due to connection rerouting while we are still
connected to server. This error will not occur on new connections. Let us try
with Read/Write connections:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal">
<b style="mso-bidi-font-weight: normal;">Read/Write Instance:<o:p></o:p></b></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;"><br /></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">C:\Users\rathish.kumar>mysql -u root -h
localhost -P6446 -p<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Enter password: *<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Welcome to the MySQL monitor.<span style="mso-spacerun: yes;"> </span>Commands end with ; or \g.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Your MySQL connection id is 32<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Server version: 8.0.11 MySQL Community Server -
GPL<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Copyright (c) 2000, 2018, Oracle and/or its affiliates.
All rights reserved. Oracle is a registered trademark of Oracle Corporation
and/or its affiliates. Other names may be trademarks of their respective owners.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">Type 'help;' or '\h' for help. Type '\c' to
clear the current input statement.<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql> select @@port;<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">| @@port |<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">|<span style="mso-spacerun: yes;"> </span>3311
|<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">+--------+<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">1 row in set (0.00 sec)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-align: justify; text-justify: inter-ideograph;">
<span class="token"><span style="background: #f8f8f8; border: none 1.0pt; color: #a67f59; font-family: "consolas"; padding: 0cm;">mysql><o:p></o:p></span></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
There is no changes required from applications, the InnoDB
Cluster will identify the changes and automatically configure itself and high
availability achieved with the help of MySQL Router.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
I suggest you to test InnoDB Cluster on lab environment and
share your findings on comment section for other readers. I will be coming with
other articles on working with InnoDB Cluster and Troubleshooting InnoDB
Cluster. Need of any assistance on InnoDB Cluster, please share it on comment
section.<o:p></o:p></div>
<div style="text-align: justify;">
<br /></div>
</div>
</div>
Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com2tag:blogger.com,1999:blog-2027709621250502166.post-38324454054057363322017-12-26T20:33:00.000+05:302017-12-26T20:47:20.351+05:30How to find database and table size in MySQL?<div dir="ltr" style="text-align: left;" trbidi="on">As a Database Administrator(DBA), your job want you to know the most atomic details of databases in your server. It happens for me many times, my boss/ delivery manager asking me, what is the size of a specific database or specific table, in this kind of situation, producing the right data will help make right decision. From my experience, I understood, it is always better to say, I will give you data in few minutes, instead of producing the incorrect data, which I had been doing for a long time.<br />
<br />
This post is about identifying the size of a database(s) or table(s). The simple script, I have been using it for quite a long, if not wrong when I started my career as DBA. You could have probably seen this/similar script on other forums as well and there are many other methods too. I am reproducing this handy script here to get work done.<br />
<div><br />
</div><!-- Google Ads - Start --><br />
<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<!-- cstechie_main_Blog1_1x1_as --><br />
<ins class="adsbygoogle" style="display:block" data-ad-client="ca-pub-9474984742004767" data-ad-slot="6299390535" data-ad-format="auto"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google Ads - End --><br />
<div>Size of a specific table:</div><div><br />
</div><div><pre class="lang-sql prettyprint prettyprinted" style="background-color: #eff0f1; border: 0px; font-stretch: inherit; line-height: inherit; margin-bottom: 1em; max-height: 600px; overflow: auto; padding: 5px; vertical-align: baseline; width: auto; word-wrap: normal;"><span style="color: #101094; font-family: "consolas" , "menlo" , "monaco" , "lucida console" , "liberation mono" , "dejavu sans mono" , "bitstream vera sans mono" , "courier new" , monospace , sans-serif;">select table_name as "Table Name",
sum(data_length+index_length)/1024/1024 as "Table Size in MB"
from information_schema.tables
where table_schema = 'Database Name' and
table_name = 'Table Name';</span><span style="color: #393318; font-family: "consolas" , "menlo" , "monaco" , "lucida console" , "liberation mono" , "dejavu sans mono" , "bitstream vera sans mono" , "courier new" , monospace , sans-serif; font-size: 13px;">
</span></pre><div>Size of a specific database:</div></div><div><br />
</div><div><pre class="lang-sql prettyprint prettyprinted" style="background-color: #eff0f1; border: 0px; font-stretch: inherit; line-height: inherit; margin-bottom: 1em; max-height: 600px; overflow: auto; padding: 5px; vertical-align: baseline; width: auto; word-wrap: normal;"><span style="color: #101094; font-family: "consolas" , "menlo" , "monaco" , "lucida console" , "liberation mono" , "dejavu sans mono" , "bitstream vera sans mono" , "courier new" , monospace , sans-serif;">select table_schema as "Database Name",
sum(data_length+index_length)/1024/1024 as "Database Size in MB"
from information_schema.tables
where table_schema = 'Database Name';</span></pre></div><br />
<!-- Google Ads - Start --><br />
<script async="" src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<!-- Rectangle Ad --><br />
<ins class="adsbygoogle" data-ad-client="ca-pub-9474984742004767" data-ad-slot="5768789207" style="display: inline-block; height: 280px; width: 336px;"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google Ads - End --><br />
<br />
<div>Size of all tables in a database with descending order:</div><div><br />
</div><div><pre class="lang-sql prettyprint prettyprinted" style="background-color: #eff0f1; border: 0px; font-stretch: inherit; line-height: inherit; margin-bottom: 1em; max-height: 600px; overflow: auto; padding: 5px; vertical-align: baseline; width: auto; word-wrap: normal;"><span style="color: #101094; font-family: "consolas" , "menlo" , "monaco" , "lucida console" , "liberation mono" , "dejavu sans mono" , "bitstream vera sans mono" , "courier new" , monospace , sans-serif;">select table_name as "Table Name",
sum(data_length+index_length)/1024/1024 as "Table Size in MB"
from information_schema.tables
where table_schema = 'Database Name'
group by table_name
order by 2 desc;</span>
</pre></div><div><br />
</div><div><div>Size of all databases in descending order:</div></div><div><br />
</div><div><pre class="lang-sql prettyprint prettyprinted" style="background-color: #eff0f1; border: 0px; font-stretch: inherit; line-height: inherit; margin-bottom: 1em; max-height: 600px; overflow: auto; padding: 5px; vertical-align: baseline; width: auto; word-wrap: normal;"><span style="color: #101094; font-family: "consolas" , "menlo" , "monaco" , "lucida console" , "liberation mono" , "dejavu sans mono" , "bitstream vera sans mono" , "courier new" , monospace , sans-serif;">select table_schema as "Database Name",
sum(data_length+index_length)/1024/1024 as "Database Size in MB"
from information_schema.tables
group by table_schema
order by 2 desc;</span></pre></div><!-- Google Ads - Start --><br />
<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<!-- cstechie_main_Blog1_1x1_as --><br />
<ins class="adsbygoogle" style="display:block" data-ad-client="ca-pub-9474984742004767" data-ad-slot="6299390535" data-ad-format="auto"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google Ads - End --><br />
<div><div>This script is enough for us to get things done. I am providing here the table description of the information_schema.tables for more understanding. The table description can be displayed by running DESCRIBE command. </div></div><div><br />
</div><div><pre class="lang-sql prettyprint prettyprinted" style="background-color: #eff0f1; border: 0px; font-stretch: inherit; line-height: inherit; margin-bottom: 1em; max-height: 600px; overflow: auto; padding: 5px; vertical-align: baseline; width: auto; word-wrap: normal;"><span style="color: #101094; font-family: "consolas" , "menlo" , "monaco" , "lucida console" , "liberation mono" , "dejavu sans mono" , "bitstream vera sans mono" , "courier new" , monospace , sans-serif;">mysql> desc information_schema.tables;
+-----------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+---------------------+------+-----+---------+-------+
| TABLE_CATALOG | varchar(512) | NO | | | |
| TABLE_SCHEMA | varchar(64) | NO | | | |
| TABLE_NAME | varchar(64) | NO | | | |
| TABLE_TYPE | varchar(64) | NO | | | |
| ENGINE | varchar(64) | YES | | NULL | |
| VERSION | bigint(21) unsigned | YES | | NULL | |
| ROW_FORMAT | varchar(10) | YES | | NULL | |
| TABLE_ROWS | bigint(21) unsigned | YES | | NULL | |
| AVG_ROW_LENGTH | bigint(21) unsigned | YES | | NULL | |
| DATA_LENGTH | bigint(21) unsigned | YES | | NULL | |
| MAX_DATA_LENGTH | bigint(21) unsigned | YES | | NULL | |
| INDEX_LENGTH | bigint(21) unsigned | YES | | NULL | |
| DATA_FREE | bigint(21) unsigned | YES | | NULL | |
| AUTO_INCREMENT | bigint(21) unsigned | YES | | NULL | |
| CREATE_TIME | datetime | YES | | NULL | |
| UPDATE_TIME | datetime | YES | | NULL | |
| CHECK_TIME | datetime | YES | | NULL | |
| TABLE_COLLATION | varchar(32) | YES | | NULL | |
| CHECKSUM | bigint(21) unsigned | YES | | NULL | |
| CREATE_OPTIONS | varchar(255) | YES | | NULL | |
| TABLE_COMMENT | varchar(2048) | NO | | | |
+-----------------+---------------------+------+-----+---------+-------+
21 rows in set (0.00 sec)</span></pre></div><div></div><!-- Google Ads - Start --><br />
<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<!-- cstechie_main_Blog1_1x1_as --><br />
<ins class="adsbygoogle" style="display:block" data-ad-client="ca-pub-9474984742004767" data-ad-slot="6299390535" data-ad-format="auto"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google Ads - End --><br />
<div><div>I hope this post will help you to complete your task quickly. Please write your comments on this post and let me know, if there are other simple methods to achieve this task. Thanks for your time.</div></div><div><br />
</div></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com25tag:blogger.com,1999:blog-2027709621250502166.post-45186840617674714462017-12-01T18:17:00.000+05:302017-12-01T18:26:12.460+05:30Deleting huge number of records in MySQL<div dir="ltr" style="text-align: left;" trbidi="on"><div style="text-align: justify;">This is a short post about DELETE data from huge table in MySQL. Most of us experienced, deleting huge record from MySQL tables taking long time, sometimes hours to complete deleting millions of records. Also, on production servers it locks the other table operations as well. Recently, I deleted around 70 million record from a production database in less than an hour. There are multiple workarounds to do this, however I am writing about the two methods which are frequently used by me for this operation. </div><div style="text-align: justify;"></div><ul><li style="text-align: justify;">Using intermediate table.</li>
<li style="text-align: justify;">Delete data in small chunks.</li>
</ul><div style="text-align: justify;">Before we proceed with using any of these methods, make sure the table has required indexes on where clause and you have a copy of the table as backup.</div><div style="text-align: justify;"><br />
</div><h4 style="text-align: justify;">Using intermediate table:</h4><div style="text-align: justify;">In this method, create a new table with similar data structure and copy only required data. <span style="text-align: left;">Rename the original table as archive or backup table and </span>Rename the new table as original table.</div><div style="text-align: justify;"><br />
</div><!-- Google Ads - Start --><br />
<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<!-- Rectangle Ad --><br />
<ins class="adsbygoogle"
style="display:inline-block;width:336px;height:280px"
data-ad-client="ca-pub-9474984742004767"
data-ad-slot="5768789207"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google Ads - End --><br />
<br />
<div style="text-align: justify;"><b>Example:</b></div><br />
<div style="text-align: justify;">Table `tabA` has 10 million record and you need only 1 million records of last 1 month. So the model query will be as follows:</div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;"><pre class="lang-sql prettyprint prettyprinted" style="background-color: #eff0f1; border: 0px; color: #393318; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-size: 13px; font-stretch: inherit; font-variant-numeric: inherit; line-height: inherit; margin-bottom: 1em; max-height: 600px; overflow: auto; padding: 5px; text-align: justify; vertical-align: baseline; width: auto; word-wrap: normal;"><code style="border: 0px; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline; white-space: inherit;"><span class="kwd" style="border: 0px; color: #101094; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">CREATE</span><span class="pln" style="border: 0px; color: #303336; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;"> </span><span class="kwd" style="border: 0px; color: #101094; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">TABLE `tabA_copy` as select * from `tabA` where predicate_col > curdate() - interval 30 day;</span></code></pre></div><div style="text-align: justify;">Now the table tabA_copy will have only the required data.</div><div style="text-align: justify;"><br />
</div><div style="text-align: justify;"><pre class="lang-sql prettyprint prettyprinted" style="background-color: #eff0f1; border: 0px; color: #393318; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-size: 13px; font-stretch: inherit; font-variant-numeric: inherit; line-height: inherit; margin-bottom: 1em; max-height: 600px; overflow: auto; padding: 5px; text-align: justify; vertical-align: baseline; width: auto; word-wrap: normal;"><code style="border: 0px; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline; white-space: inherit;"><span class="kwd" style="border: 0px; color: #101094; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Rename table tabA to tabA_archive;</span></code></pre></div><div style="text-align: justify;"><pre class="lang-sql prettyprint prettyprinted" style="background-color: #eff0f1; border: 0px; color: #393318; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-size: 13px; font-stretch: inherit; font-variant-numeric: inherit; line-height: inherit; margin-bottom: 1em; max-height: 600px; overflow: auto; padding: 5px; text-align: justify; vertical-align: baseline; width: auto; word-wrap: normal;"><code style="border: 0px; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline; white-space: inherit;"><span class="kwd" style="border: 0px; color: #101094; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">Rename table tabA_copy to tabA;</span></code></pre></div><div style="text-align: justify;"><span style="background-color: white;">This method is useful when you wanted to copy only small percentage of the huge table data. Since, moving small portion of data to new table is cheaper when compared to delete huge data.</span></div><div style="text-align: justify;"><span style="background-color: white;"><br />
</span></div><br />
<!-- Google Ads - Start --><br />
<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<!-- cstechie_main_Blog1_1x1_as --><br />
<ins class="adsbygoogle"
style="display:block"
data-ad-client="ca-pub-9474984742004767"
data-ad-slot="6299390535"
data-ad-format="auto"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google Ads - End --><br />
<br />
<h4 style="text-align: justify;"><span style="background-color: white;">Delete data in smaller chunks:</span></h4><div style="text-align: justify;"><span style="background-color: white;">In this method, data will be deleted as smaller chunks. Even in case of rollback also, it is cheaper to rollback smaller data sets. I have been using the following procedure to implement this method. Actually, it is a dynamic procedure with PREPARED statement to satisfy all the requirements. For simplicity, I am writing the core part of it. </span></div><div style="text-align: justify;"><span style="background-color: white;"><br />
</span></div><div style="text-align: justify;"><pre class="lang-sql prettyprint prettyprinted" style="background-color: #eff0f1; border: 0px; color: #393318; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-size: 13px; font-stretch: inherit; font-variant-numeric: inherit; line-height: inherit; margin-bottom: 1em; max-height: 600px; overflow: auto; padding: 5px; vertical-align: baseline; width: auto; word-wrap: normal;"><code style="border: 0px; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline; white-space: inherit;"><span class="kwd" style="border: 0px; color: #101094; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">drop procedure if exists huge_delete;
delimiter //
create procedure huge_delete()
begin
</span> <span class="kwd" style="border: 0px; color: #101094; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">declare rows int;
</span> <span class="kwd" style="border: 0px; color: #101094; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">declare rows_deleted int;
</span> <span class="kwd" style="border: 0px; color: #101094; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">set rows = 1;
</span> <span class="kwd" style="border: 0px; color: #101094; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">set rows_deleted = 10000;
</span> <span class="kwd" style="border: 0px; color: #101094; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">while rows > 0
</span> <span class="kwd" style="border: 0px; color: #101094; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">do
delete from db.tabA where predicate_col < curdate() - interval 90 day order by `id` limit 10000;
set rows = row_count();
set rows_deleted = rows_deleted + row_count();
select rows_deleted as "Rows Deleted";
</span> <span class="kwd" style="border: 0px; color: #101094; font-family: inherit; font-size: inherit; font-stretch: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; line-height: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">end while;
end //
delimiter ;</span></code></pre></div><!-- Google Ads - Start --><br />
<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<!-- cstechie_main_Blog1_1x1_as --><br />
<ins class="adsbygoogle"
style="display:block"
data-ad-client="ca-pub-9474984742004767"
data-ad-slot="6299390535"
data-ad-format="auto"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google Ads - End --><br />
<div style="text-align: justify;"><span style="background-color: white;">This method is suited when you wanted delete huge set of data (in millions) from the table. Make sure, you have index on the where clause column.</span></div><div style="text-align: justify;"><span style="background-color: white;"><b><br />
</b></span></div><div style="text-align: justify;"><span style="background-color: white;"><b>Note:</b></span></div><div style="text-align: justify;"><span style="background-color: white;"></span><br />
<div><span style="background-color: white;">I am using innodb_flush_log_trx_commit = 2 to reduce the total response time. There are certain points to be considered before setting the innodb_flush_log_at_trx_commit as 2. Take a look at <a href="http://www.rathishkumar.in/2017/10/understanding-mysql-innodb-flush-log-at-trx-commit.html" rel="nofollow" target="_blank">this article</a> before you proceed with this value.</span></div><span style="background-color: white;"> </span> <br />
<div><span style="background-color: white;"><br />
</span></div><span style="background-color: white;"> </span> <div><span style="background-color: white;">This setting runs with autocommit = 1 value which is default. In case if you run with transactions, make sure you commit the transaction at every iteration.</span><br />
<span style="background-color: white;"><br />
</span></div><span style="background-color: white;"> <h4>Caution:</h4><div><div>If you are running on Replication master server make sure, slave is in synchronization with master. In my recent implementation, the slave is very long time behind the master. I fixed it by skipping some transactions and later used compare utility to make it sync.</div><div><br />
</div><div>I recommend you to test this steps and confirm that, it is working as you expected before using it on production server. I hope this post will help you to get the things done!.</div></div><!-- Google Ads - Start --><br />
<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<!-- cstechie_main_Blog1_1x1_as --><br />
<ins class="adsbygoogle"
style="display:block"
data-ad-client="ca-pub-9474984742004767"
data-ad-slot="6299390535"
data-ad-format="auto"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google Ads - End --><br />
</span></div></div>Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com66tag:blogger.com,1999:blog-2027709621250502166.post-89667525676159541812017-10-13T17:38:00.002+05:302018-03-02T13:29:48.446+05:30Understanding MySQL innodb_flush_log_at_trx_commit variable<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="MsoNormal" style="text-align: justify;">
The main objective of this post is to understand why innodb_fush_log_at_trx_commit variable is used? What are the applicable values for innodb_fush_log_at_trx_commit? How innodb_fush_log_at_trx_commit value will impact the MySQL performance and data safety? How to change innodb_fush_log_at_trx_commit value? and How to change the frequency of InnoDB log flush?<o:p></o:p></div>
<br />
<div class="MsoNormal" style="text-align: justify;">
</div>
<h4 style="text-align: justify;">
Why innodb_fush_log_at_trx_commit?</h4>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<br />
<div class="MsoNormal" style="text-align: justify;">
In order to understand why we need innodb_fush_log_at_trx_commit variable, we should know about how InnoDB works. It is a huge and complex topic and it is not the scope of this article. I am trying to cover this topic in simple words and it is given below:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br />
<ol>
<li>InnoDB performs most of its operations at the memory (InnoDB Buffer Pool)</li>
<li>It will write all changes from memory to the transaction log (InnoDB Log File)</li>
<li>From transaction log - it will flush (write) data to storage disk (durable storage)</li>
</ol>
</div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
In order to achieve durability of data, we need to store each and every transaction data into hard disk storage. But consider, in a busy system, for each transaction commit, if InnoDB trying to flush (write) data to slow running disk, what will happen?, So how do we manage this situation, where we need to store each transaction data and at the same time maintaining good performance of the system. <o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
The InnoDB provides the solution for this situation, based on your system, you can tell InnoDB, when you want to flush (write) data to disk. For example, you can tell InnoDB to work as mentioned below:<br />
<br />
<ol>
<li>Write to log file and flush data to disk in specified interval, not for every transaction commit.</li>
<li>Write to log and flush to disk for each transaction commit.</li>
<li>Write to log for every transaction commit but flush to disk at an interval not for each transaction commit.</li>
</ol>
</div>
<div class="MsoListParagraphCxSpLast" style="mso-list: l0 level1 lfo1; text-align: justify; text-indent: -18.0pt;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<!-- Google - Ad Starts --><br />
<script async="" src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<!-- Responsive ad --><br />
<ins class="adsbygoogle" data-ad-client="ca-pub-9474984742004767" data-ad-format="auto" data-ad-slot="8520082937" style="display: block;"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google - Ad Ends --><br />
<div class="MsoNormal" style="text-align: justify;">
Based on the application standard you can adjust these settings to maintain balance between performance and data safety. InnoDB provides a configurable variable to achieve this balance, this variable is called innodb_fush_log_at_trx_commit. This variable controls how often the log buffer is flushed.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<h4 style="text-align: justify;">
What are the applicable values for innodb_fush_log_at_trx_commit and its impact?</h4>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
InnoDB supports following 3 values for innodb_fush_log_at_trx_commit variable. Let see how these variables will change flush to disk behaviour:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px; text-align: start;">innodb_flush_log_at_trx_commit</span>=0<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
When you set innodb_flush_log_trx_at_commit=0, InnoDB will write the modified data (in InnoDB Buffer Pool) to log file (ib_logfile) and flush the log file (write to disk) every second, but it will not do anything at transaction commit.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Here, in case if there is a power failure or system crash, all the unflushed data will not be recoverable, since, it is not either written to log file or stored disk. <o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px; text-align: start;">innodb_flush_log_at_trx_commit</span>=1<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
When you set innodb_flush_log_trx_commit=1, InnoDB will write the log buffer to transaction log and flush to durable storage for every transaction.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Here, for all transaction commit, InnoDB will write to log and then write to disk, if in case the slower disk storage, it will badly impact the performance, i.e. the number of InnoDB transaction per second will be reduced.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 12.8px; text-align: start;">innodb_flush_log_at_trx_commit</span>=2<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
When you set innodb_flush_log_trx_commit = 2, InnoDB will write the log buffer to log file at every commit, but don't write data to disk. InnoDB flushes data once in every second. <o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
Option 2, even if there is a power failure or system crash, data will be available in log file and can be recoverable.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
</div>
<!-- Google - Ad Start --><br />
<script async="" src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<!-- Rectangle Ad --><br />
<ins class="adsbygoogle" data-ad-client="ca-pub-9474984742004767" data-ad-slot="5768789207" style="display: inline-block; height: 280px; width: 336px;"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google - Ad End --><br />
<h4 style="text-align: justify;">
So which option to be used on your application?</h4>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
We have seen, the behaviour of each values, now it is based on the application requirement, we need to choose this variable value.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
If performance is the main goal, you can set the value as 2. Since, InnoDB writing to disk once in every second, not for every transaction commit and it will improve the performance dramatically. In case if there is power failure or crash, data can be recoverable from transaction log.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
If data safety is the main goal, you can set the value as 1, so that for every transaction commit, InnoDB will flush to disk. But performance may reduce in this case. <o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
If you set value as 0, InnoDB will write from buffer to log once in every second and it will not perform any flush to disk operation for every transaction commit. The problem with this option is if in case of any power failure or system crash, there may be a chance of losing data up to one second.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<h4 style="text-align: justify;">
Key points:</h4>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
During buffer to log operation, i.e. writing from InnoDB buffer pool to InnoDB transaction log file - data is simply moved from InnoDB buffer to Operating system's cache, actually not written to the durable storage. So if you set innodb_fush_log_at_trx_commit either 0 or 2, there is a possibility of losing data up to one second.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
If innodb_fush_log_at_trx_commit is set to 1 - InnoDB compels Operating System to flush data to durable storage. Writing to disk is a slow process and it is I/O blocking operation to ensure data is written to disk. So using this option, there is a chance; number of transaction per second will be reduced.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Note that, by default MySQL will auto commit the transactions.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Setting innodb_fush_log_at_trx_commit as 2 is useful when restoring huge databases, it will reduce the restoration time dramatically, and there are different opinions on this point, it is better to test in our own hand. In my experience, it really helped with the reduced restoration time.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Data Definition Language (DDL) changes flush the InnoDB log, independent of the innodb_fush_log_at_trx_commit setting. <o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Crash recovery operation works regardless of the innodb_fush_log_at_trx_commit setting, I am writing another article on InnoDB crash recovery.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<!-- Google - Ad Start --><br />
<script async="" src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<!-- cstechie_main_Blog1_1x1_as --><br />
<ins class="adsbygoogle" data-ad-client="ca-pub-9474984742004767" data-ad-format="auto" data-ad-slot="6299390535" style="display: block;"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google - Ad End --><br />
<h4 style="text-align: justify;">
How to configure innodb_fush_log_at_trx_commit variable?</h4>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
The scope of the innodb_fush_log_at_trx_commit is GLOBAL and it can be set at dynamically without restarting server.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Dynamically on command line you can set innodb_fush_log_at_trx_commit as follows:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
SET GLOBAL innodb_fush_log_at_trx_commit = 2;<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
On configuration file, you can set it as follows:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<blockquote class="tr_bq" style="text-align: justify;">
[mysqld]<br />
innodb_fush_log_at_trx_commit = 2</blockquote>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
Note: It requires server restart. Before making changes to configuration file, analyse the impact by setting it on dynamically.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
If you try to set innodb_fush_log_at_trx_commit as session level variables, you will encounter the following error:<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<blockquote class="tr_bq" style="text-align: justify;">
mysql> set session innodb_fush_log_at_trx_commit = 2;<br />
ERROR 1229 (HY000): Variable 'innodb_fush_log_at_trx_commit' is a GLOBAL variable and should be set with SET GLOBAL<br />
mysql></blockquote>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<h4 style="text-align: justify;">
How to change the frequency of InnoDB log file flushing?</h4>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
The variable innodb_flush_log_at_timeout controls the frequency in which InnoDB flush log files. The frequency ranges from 1 to 2700 seconds, with the default value 1.</div>
<div class="MsoNormal" style="text-align: justify;">
<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<!-- Google - Ad - Starts --><br />
<script async="" src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script><br />
<!-- cstechie_main_Blog1_1x1_as --><br />
<ins class="adsbygoogle" data-ad-client="ca-pub-9474984742004767" data-ad-format="auto" data-ad-slot="6299390535" style="display: block;"></ins><br />
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script><br />
<!-- Google - Ad - End --><br /></div>
<div class="MsoNormal" style="text-align: justify;">
Note that, the higher this number, there is a higher the chance of losing data, in case of power failure or system crash.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
For example, if you set this value as 5 seconds, in case of power failure, you may lose data upto 5 seconds.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
In replication topology, to maintain durability and consistency of data, you can leave the default value, i.e. innodb_fush_log_at_trx_commit = 1.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
I hope this article help you to understand, how InnoDB flush data and how it impact MySQL performance and data safety, also, how you can configure this variable to achieve maximum benefit. </div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
In future article let us see about how InnoDB crash recovery works and its configuration settings. If I missed something or if you wish to share your thoughts on this article, please mention in comment section, I will edit the post after review.<o:p></o:p></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
<br />
<div class="MsoNormal" style="text-align: justify;">
<br /></div>
</div>
Rathishhttp://www.blogger.com/profile/11359580686405655070noreply@blogger.com24