Skip to content

Commit c8daaa5

Browse files
shubha-rajanbusunkim96kurtisvg
authored
Add sample for connecting to SQL Server with SQLAlchemy/pyodbc (#3135)
* wrote Python sample for SQL Server and corresponding readme * configured app.yaml and Dockerfile to deploy to GAE flex * update readme * update region tags * fixed linting errors * Update copyright year Co-Authored-By: Bu Sun Kim <[email protected]> * Update cloud-sql/sql-server/sqlalchemy/app.yaml Co-Authored-By: Kurtis Van Gent <[email protected]> * Update cloud-sql/sql-server/sqlalchemy/templates/index.html Co-Authored-By: Kurtis Van Gent <[email protected]> * Update cloud-sql/sql-server/sqlalchemy/main.py Co-Authored-By: Kurtis Van Gent <[email protected]> * Apply suggestions from code review Co-Authored-By: Kurtis Van Gent <[email protected]> * addressed PR comments and reworked Dockerfile to use debian image * Update cloud-sql/sql-server/sqlalchemy/Dockerfile Co-Authored-By: Kurtis Van Gent <[email protected]> * Update cloud-sql/sql-server/sqlalchemy/Dockerfile Co-Authored-By: Kurtis Van Gent <[email protected]> * fix syntax error in Dockerfile * updated region tags * add requirements-test.txt Co-authored-by: Bu Sun Kim <[email protected]> Co-authored-by: Kurtis Van Gent <[email protected]>
1 parent 0db5c2e commit c8daaa5

File tree

9 files changed

+448
-0
lines changed

9 files changed

+448
-0
lines changed
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# This file specifies files that are *not* uploaded to Google Cloud Platform
2+
# using gcloud. It follows the same syntax as .gitignore, with the addition of
3+
# "#!include" directives (which insert the entries of the given .gitignore-style
4+
# file at that point).
5+
#
6+
# For more information, run:
7+
# $ gcloud topic gcloudignore
8+
#
9+
.gcloudignore
10+
# If you would like to upload your .git directory, .gitignore file or files
11+
# from your .gitignore file, remove the corresponding line
12+
# below:
13+
.git
14+
.gitignore
15+
16+
# Python pycache:
17+
__pycache__/
18+
# Ignored by the build system
19+
/setup.cfg
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
__pycache__
2+
.pytest_cache
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Copyright 2020 Google, LLC.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# Use the official Python image.
16+
# https://hub.docker.com/_/python
17+
FROM python:3.8-buster
18+
19+
#Download the desired package(s) for Microsoft ODBC 17 Driver
20+
RUN (curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -) && \
21+
curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list && \
22+
apt-get update && ACCEPT_EULA=Y apt-get install -y msodbcsql17 unixodbc-dev
23+
24+
# Copy application dependency manifests to the container image.
25+
# Copying this separately prevents re-running pip install on every code change.
26+
COPY requirements.txt ./
27+
28+
# Install production dependencies.
29+
RUN pip install -r requirements.txt && \
30+
pip install gunicorn
31+
32+
# Copy local code to the container image.
33+
ENV APP_HOME /app
34+
WORKDIR $APP_HOME
35+
COPY . ./
36+
37+
38+
# Run the web service on container startup. Here we use the gunicorn
39+
# webserver, with one worker process and 8 threads.
40+
# For environments with multiple CPU cores, increase the number of workers
41+
# to be equal to the cores available.
42+
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 main:app
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Connecting to Cloud SQL - SQL Server
2+
3+
## Before you begin
4+
5+
1. If you haven't already, set up a Python Development Environment by following the [python setup guide](https://cloud.google.com/python/setup) and
6+
[create a project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#creating_a_project).
7+
8+
1. [Create a Google Cloud SQL "SQL Server" instance](
9+
https://console.cloud.google.com/sql/choose-instance-engine).
10+
11+
6. Under the instance's "USERS" tab, create a new user. Note the "User name" and "Password".
12+
13+
7. Create a new database in your Google Cloud SQL instance.
14+
15+
1. List your database instances in [Cloud Cloud Console](
16+
https://console.cloud.google.com/sql/instances/).
17+
18+
2. Click your Instance Id to see Instance details.
19+
20+
3. Click DATABASES.
21+
22+
4. Click **Create database**.
23+
24+
2. For **Database name**, enter `votes`.
25+
26+
3. Click **CREATE**.
27+
28+
1. Install the version of [Microsoft ODBC 17 Driver for SQL Server](https://docs.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver15_) for your operating system.
29+
30+
1. Create a service account with the 'Cloud SQL Client' permissions by following these
31+
[instructions](https://cloud.google.com/sql/docs/postgres/connect-external-app#4_if_required_by_your_authentication_method_create_a_service_account).
32+
Download a JSON key to use to authenticate your connection.
33+
34+
1. Use the information noted in the previous steps:
35+
```bash
36+
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service/account/key.json
37+
export CLOUD_SQL_CONNECTION_NAME='<MY-PROJECT>:<INSTANCE-REGION>:<INSTANCE-NAME>'
38+
export DB_USER='my-db-user'
39+
export DB_PASS='my-db-pass'
40+
export DB_NAME='my_db'
41+
```
42+
Note: Saving credentials in environment variables is convenient, but not secure - consider a more
43+
secure solution such as [Cloud KMS](https://cloud.google.com/kms/) to help keep secrets safe.
44+
45+
## Running locally
46+
47+
To run this application locally, download and install the `cloud_sql_proxy` by
48+
following the instructions [here](https://cloud.google.com/sql/docs/mysql/sql-proxy#install).
49+
50+
Then, use the following command to start the proxy in the
51+
background using TCP:
52+
```bash
53+
./cloud_sql_proxy -instances=${CLOUD_SQL_CONNECTION_NAME}=tcp:1433 sqlserver -u ${DB_USER} --host 127.0.0.1
54+
```
55+
56+
Next, setup install the requirements into a virtual enviroment:
57+
```bash
58+
virtualenv --python python3 env
59+
source env/bin/activate
60+
pip install -r requirements.txt
61+
```
62+
63+
Finally, start the application:
64+
```bash
65+
python main.py
66+
```
67+
68+
Navigate towards `http://127.0.0.1:8080` to verify your application is running correctly.
69+
70+
## Deploy to App Engine Flexible
71+
72+
App Engine Flexible supports connecting to your SQL Server instance through TCP
73+
74+
First, update `app.yaml` with the correct values to pass the environment
75+
variables and instance name into the runtime.
76+
77+
Then, make sure that the service account `service-{PROJECT_NUMBER}>@gae-api-prod.google.com.iam.gserviceaccount.com` has the IAM role `Cloud SQL Client`.
78+
79+
Next, the following command will deploy the application to your Google Cloud project:
80+
```bash
81+
gcloud beta app deploy
82+
```
83+
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Copyright 2020 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
runtime: custom
16+
env: flex
17+
18+
# Remember - storing secrets in plaintext is potentially unsafe. Consider using
19+
# something like https://cloud.google.com/secret-manager/docs to help keep secrets secret.
20+
env_variables:
21+
DB_USER: <your-username>
22+
DB_PASS: <your-password>
23+
DB_NAME: <your-db-name>
24+
# Whether the app is deployed or running locally
25+
DEPLOYED: true
26+
27+
beta_settings:
28+
cloud_sql_instances: <project-name>:<region-name>:<instance-name>=tcp:1433
Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
# Copyright 2020 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
import datetime
16+
import logging
17+
import os
18+
19+
from flask import Flask, render_template, request, Response
20+
import sqlalchemy
21+
from sqlalchemy import Table
22+
from sqlalchemy import Column
23+
from sqlalchemy import DateTime, Integer, String
24+
25+
app = Flask(__name__)
26+
27+
logger = logging.getLogger()
28+
29+
# [START cloud_sql_server_sqlalchemy_create]
30+
# Remember - storing secrets in plaintext is potentially unsafe. Consider using
31+
# something like https://cloud.google.com/kms/ to help keep secrets secret.
32+
db_user = os.environ.get("DB_USER")
33+
db_pass = os.environ.get("DB_PASS")
34+
db_name = os.environ.get("DB_NAME")
35+
36+
# When deployed to GAE Flex for TCP, use "172.17.0.1" to connect
37+
host = "172.17.0.1" if os.environ.get("DEPLOYED") else "127.0.0.1"
38+
39+
# The SQLAlchemy engine will help manage interactions, including automatically
40+
# managing a pool of connections to your database
41+
db = sqlalchemy.create_engine(
42+
# Equivalent URL:
43+
# mssql+pyodbc://<db_user>:<db_pass>@/<host>:<port>/<db_name>?driver=ODBC+Driver+17+for+SQL+Server
44+
sqlalchemy.engine.url.URL(
45+
"mssql+pyodbc",
46+
username=db_user,
47+
password=db_pass,
48+
database=db_name,
49+
host=host,
50+
port=1433,
51+
query={"driver": "ODBC Driver 17 for SQL Server"},
52+
),
53+
# ... Specify additional properties here.
54+
# [START_EXCLUDE]
55+
# [START cloud_sql_server_sqlalchemy_limit]
56+
# Pool size is the maximum number of permanent connections to keep.
57+
pool_size=5,
58+
# Temporarily exceeds the set pool_size if no connections are available.
59+
max_overflow=2,
60+
# The total number of concurrent connections for your application will be
61+
# a total of pool_size and max_overflow.
62+
# [END cloud_sql_server_sqlalchemy_limit]
63+
# [START cloud_sql_server_sqlalchemy_backoff]
64+
# SQLAlchemy automatically uses delays between failed connection attempts,
65+
# but provides no arguments for configuration.
66+
# [END cloud_sql_server_sqlalchemy_backoff]
67+
# [START cloud_sql_server_sqlalchemy_timeout]
68+
# 'pool_timeout' is the maximum number of seconds to wait when retrieving a
69+
# new connection from the pool. After the specified amount of time, an
70+
# exception will be thrown.
71+
pool_timeout=30, # 30 seconds
72+
# [END cloud_sql_server_sqlalchemy_limit]
73+
# [START cloud_sql_server_sqlalchemy_lifetime]
74+
# 'pool_recycle' is the maximum number of seconds a connection can persist.
75+
# Connections that live longer than the specified amount of time will be
76+
# reestablished
77+
pool_recycle=1800, # 30 minutes
78+
# [END cloud_sql_server_sqlalchemy_lifetime]
79+
echo=True # debug
80+
# [END_EXCLUDE]
81+
)
82+
# [END cloud_sql_server_sqlalchemy_create]
83+
84+
85+
@app.before_first_request
86+
def create_tables():
87+
# Create tables (if they don't already exist)
88+
if not db.has_table("votes"):
89+
metadata = sqlalchemy.MetaData(db)
90+
Table(
91+
"votes",
92+
metadata,
93+
Column("vote_id", Integer, primary_key=True, nullable=False),
94+
Column("time_cast", DateTime, nullable=False),
95+
Column("candidate", String(6), nullable=False),
96+
)
97+
metadata.create_all()
98+
99+
100+
@app.route("/", methods=["GET"])
101+
def index():
102+
votes = []
103+
with db.connect() as conn:
104+
# Execute the query and fetch all results
105+
recent_votes = conn.execute(
106+
"SELECT TOP(5) candidate, time_cast FROM votes "
107+
"ORDER BY time_cast DESC"
108+
).fetchall()
109+
# Convert the results into a list of dicts representing votes
110+
for row in recent_votes:
111+
votes.append({"candidate": row[0], "time_cast": row[1]})
112+
113+
stmt = sqlalchemy.text(
114+
"SELECT COUNT(vote_id) FROM votes WHERE candidate=:candidate"
115+
)
116+
# Count number of votes for tabs
117+
tab_result = conn.execute(stmt, candidate="TABS").fetchone()
118+
tab_count = tab_result[0]
119+
# Count number of votes for spaces
120+
space_result = conn.execute(stmt, candidate="SPACES").fetchone()
121+
space_count = space_result[0]
122+
123+
return render_template("index.html", recent_votes=votes,
124+
tab_count=tab_count, space_count=space_count)
125+
126+
127+
@app.route("/", methods=["POST"])
128+
def save_vote():
129+
# Get the team and time the vote was cast.
130+
team = request.form["team"]
131+
time_cast = datetime.datetime.utcnow()
132+
# Verify that the team is one of the allowed options
133+
if team != "TABS" and team != "SPACES":
134+
logger.warning(team)
135+
return Response(response="Invalid team specified.", status=400)
136+
137+
# [START cloud_sql_server_sqlalchemy_connection]
138+
# Preparing a statement before hand can help protect against injections.
139+
stmt = sqlalchemy.text(
140+
"INSERT INTO votes (time_cast, candidate)"
141+
" VALUES (:time_cast, :candidate)"
142+
)
143+
try:
144+
# Using a with statement ensures that the connection is always released
145+
# back into the pool at the end of statement (even if an error occurs)
146+
with db.connect() as conn:
147+
conn.execute(stmt, time_cast=time_cast, candidate=team)
148+
except Exception as e:
149+
# If something goes wrong, handle the error in this section. This might
150+
# involve retrying or adjusting parameters depending on the situation.
151+
# [START_EXCLUDE]
152+
logger.exception(e)
153+
return Response(
154+
status=500,
155+
response="Unable to successfully cast vote! Please check the "
156+
"application logs for more details.",
157+
)
158+
# [END_EXCLUDE]
159+
# [END cloud_sql_server_sqlalchemy_connection]
160+
161+
return Response(
162+
status=200,
163+
response="Vote successfully cast for '{}' at time {}!".format(
164+
team, time_cast),
165+
)
166+
167+
168+
if __name__ == "__main__":
169+
app.run(host="127.0.0.1", port=8080, debug=True)
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
pytest==5.3.2
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Flask==1.1.1
2+
SQLAlchemy==1.3.13
3+
pyodbc==4.0.30
4+

0 commit comments

Comments
 (0)