GenericSuite is a versatile backend solution, designed to provide a comprehensive suite of features for Python APIs. It supports various frameworks including Chalice, FastAPI, and Flask, making it adaptable to a range of projects. This repository contains the backend scripts necessary to build and deploy APIs made by the backend version of The GenericSuite.
- AWS Deployment: Deployment to AWS as Lambda Function with API Gateway usig SAM (AWS Serverless Application Model).
- Local Development Environment: running with http or https, with or without Docker.
-
Local DNS Server: to allow https API access with a domain name like
app.exampleapp.local
and allow access from another devices locally (e.g. smartphones) to test your App. - Self-signed SSL certificates creation: to allow local development frontend and backend environments run over secure https connections.
- Common JSON config management: to add the Git Submodule with the common JSON config directories.
- Local MongoDB Docker conntainer: used by the test site and allows to have an offline local development environment.
To get started with GenericSuite, follow these steps:
Check The GenericSuite Getting Started guide for more details.
npm init
npm install -D genericsuite-be-scripts
To generate sef-signed SSL certificates, office-addin-dev-certs
is required:
npm install -D office-addin-dev-certs
Copy the Makefile
template from node_modules/genericsuite-be-scripts
:
cp node_modules/genericsuite-be-scripts/Makefile ./Makefile
Create the project's scripts/aws_big_lambda
and scripts/aws
directories and copy the templates:
make init_sam
Or...
sh node_modules/genericsuite-be-scripts/scripts/aws_big_lambda/init_sam.sh
If you're going to develop with the Chalice framework:
Create the .chalice
directory and copy the .chalice/config-example.json
template:
make init_chalice
Or...
sh node_modules/genericsuite-be-scripts/scripts/aws/init_chalice.sh
If you need to do any customization to the samconfig.toml
:
Edit the template-samconfig.toml
file:
vi scripts/aws_big_lambda/template-samconfig.toml
# or
# code scripts/aws_big_lambda/template-samconfig.toml
Check for some customization needed.
NOTE: The deployment script big_lambdas_manager.sh will replace APP_NAME_LOWERCASE_placeholder
with the Application name defined in the APP_NAME
variable in .env
file.
When you need to add new endpoints to your App:
Edit the template-sam.yml
:
vi scripts/aws_big_lambda/template-sam.yml
# or
# code scripts/aws_big_lambda/template-sam.yml
In this file is where the endpoints are defined, as well as other SAM deployment elements like the environment variables used by the AWS Lambda function. You can add new endpoints or customize it as you needed.
There's an endpoint definition template in the file node_modules/genericsuite-be-scripts/scripts/aws_big_lambda/template-sam-endpoint-entry.yml.
Be careful about elements ending with _placeholder
because they are replaced by the deployment script big_lambdas_manager.sh
with the corresponding values.
If you need to add additional environment variables to your App:
Edit the update_additional_envvars.sh
file:
vi scripts/aws/update_additional_envvars.sh
# or
# code scripts/aws/update_additional_envvars.sh
Add your additional environment variables replacements in scripts/aws/update_additional_envvars.sh
as:
perl -i -pe"s|ENVVAR_NAME_placeholder|${ENVVAR_NAME}|g" "${CONFIG_FILE}"
... replacing "ENVVAR_NAME" with the name of the environment variable
Add the additional environment variables to the .env
file:
ENVVAR_NAME=ENVVAR_VALUE
... replacing "ENVVAR_NAME" with the name of the environment variable and ENVVAR_VALUE with its value.
Add the additional environment variables to the scripts/aws_big_lambda/template-sam.yml
file, in the APIHandler > Properties > Environment > Variables
section. E.g.
.
.
APIHandler:
.
.
Properties:
.
.
Environment:
Variables:
ENVVAR_NAME: ENVVAR_VALUE
.
.
... replacing "ENVVAR_NAME" with the name of the environment variable and ENVVAR_VALUE with its value.
If you're using the Chalice framework, add the additional environment variables to the .chalice/config-example.json
file, in the main environment_variables
section. E.g.
{
"version": "2.0",
.
.
"environment_variables": {
.
.
"ENVVAR_NAME": "ENVVAR_NAME_placeholder"
},
"stages": {
.
.
... replacing "ENVVAR_NAME" with the name of the environment variable (in both places).
To start the development server for the dev
stage and a local docker MongoDB container:
make run
To start the development server for the qa
stage and MongoDB Atlas:
make run_qa
When there are changes to the dependencies or .env
file, restart the local development server:
make restart_qa
To perform a QA deployment as an AWS Lambda Function and AWS API Gateway:
make deploy_qa
- Install default package categories from Pipfile.
Runspipenv install
.
Reference: https://pipenv.pypa.io/en/latest/commands.html#install
make install
- Install both develop and default package categories from Pipfile.
Runspipenv install --dev
.
make install_dev
- Install from the Pipfile.lock and completely ignore Pipfile information.
Runspipenv install --ignore-pipfile
.
make locked_install
- Install both develop and default package categories from the Pipfile.lock and completely ignore Pipfile information.
Runspipenv install --dev --ignore-pipfile
.
make locked_dev
- Generates the
requirements.txt
file.
Runssh scripts/aws/run_aws.sh pipfile
.
make requirements
or...
make lock_pip_file
- Clean install.
Alias that runsmake clean_rm
andmake install
.
make fresh
- Alias to run
make clean_rm
,make clean_temp_dir
, andmake clean_logs
.
make clean
- Remove a virtual environment created by "pipenv run".
Runspipenv --rm
.
make clean_rm
- Clean logs (in /logs directory).
Runs shscripts/clean_logs.sh
.
make clean_logs
- Clean logs, cache and temporary files.
Runssh scripts/aws/run_aws.sh clean
.
make clean_temp_dir
- Install development tools (pyenv, pipenv, make, and optionally: poetry, saml2aws).
Check node_modules/genericsuite-be-scripts/scripts/install_dev_tools.sh for more details about how to configure via.env
file.
make install_tools
- Show ports in use.
Runssh scripts/run_lsof.sh
.
make lsof
- Start the local MongoDB docker container and run the tests.
Runssh scripts/run_app_tests.sh
.
make test
- Execute the test without starting the local MongoDB docker container.
Runssh scripts/aws/run_tests.sh
.
make test_only
- Execute Prospector.
Runspipenv run prospector
.
make lint
- Execute MyPy.
Runspipenv run mypy .
.
make types
- Execute Coverage.
Runspipenv run coverage run -m unittest discover tests
andpipenv run coverage report
.
make coverage
- Execute Yapf Formatter and PyCodeStyle.
Runspipenv run yapf -i *.py **/*.py **/**/*.py
andpycodestyle
.
References:
make format
- Execute Yapf (in "print the diff for the fixed source" mode) and PyCodeStyle.
Runspipenv run yapf --diff *.py **/*.py **/**/*.py
andpycodestyle
.
make format_check
- Perform a complete Lint, Type check, unit and integration test, format check, and styling before deployments.
Alias to runmake lint
,make types
,make tests
,make format_check
, andmake pycodestyle
.
make qa
- Start the local MongoDB docker container (used for testing and
dev
stage run).
Runssh scripts/mongo/run_mongo_docker.sh run
.
make mongo_docker
- Stop the local MongoDB docker container.
Runssh scripts/mongo/run_mongo_docker.sh down
make mongo_docker_down
- Set parameters on
.chalice/config.json
as the production stage.
Runssh scripts/aws/set_chalice_cnf.sh prod
.
make config
- Set parameters on
.chalice/config.json
with no specific stage.
Runssh scripts/aws/set_chalice_cnf.sh
.
make config_dev
- Set parameters on
.chalice/config.json
as the Development stage.
Runssh scripts/aws/set_chalice_cnf.sh mongo_docker
.
make config_local
- Set parameters on
.chalice/config.json
as the QA stage with CORS specific variables replacement, to allow use the QA live database from the local development environment.
Runssh scripts/aws/set_chalice_cnf.sh qa
.
References:-
APP_CORS_ORIGIN_QA_CLOUD
andAPP_CORS_ORIGIN_QA_LOCAL
in the .env.example file.
-
make config_qa
- Set parameters on
.chalice/config.json
to prepare the QA deployment.
Runssh scripts/aws/set_chalice_cnf.sh qa deploy
,
make config_qa_for_deployment
- Set parameters on
.chalice/config.json
as the Staging stage.
Runssh scripts/aws/set_chalice_cnf.sh staging
.
make config_staging
- Create the AWS Stack via Chalice command.
Runssh scripts/aws/run_aws.sh create_stack
.
make build
- Generates the
requirements.txt
.
Runssh scripts/aws/run_aws.sh pipfile
.
make build_local
- Describe the AWS stack with the Chalice command.
Runssh scripts/aws/run_aws.sh describe_stack
.
make build_check
- Alias to run
make unbuild_qa
.
make unbuild
- Delete the Chalice QA App.
Runssh scripts/aws/run_aws.sh delete_app qa
.
make unbuild_qa
- Delete the Chalice Staging App.
Runssh scripts/aws/run_aws.sh delete_app staging
make unbuild_staging
- Delete the AWS stack created by Chalice.
Runssh scripts/aws/run_aws.sh delete_stack
make delete_stack
- Create a default
${HOME}/.aws/config
AWS configuration.
Runssh scripts/aws/create_aws_config.sh
.
make create_aws_config
- Create the Development S3 buckets.
Runssh scripts/aws/create_chatbot_s3_bucket.sh dev
.
make create_s3_bucket_dev
- Create the QA S3 buckets.
Runssh scripts/aws/create_chatbot_s3_bucket.sh qa
.
make create_s3_bucket_qa
- Create the Staging S3 buckets.
Runssh scripts/aws/create_chatbot_s3_bucket.sh staging
.
make create_s3_bucket_staging
- Create the Production S3 buckets.
Runssh scripts/aws/create_chatbot_s3_bucket.sh prod
.
make create_s3_bucket_prod
- Deploy the App on QA.
Runsmake create_s3_bucket_qa
,sh scripts/aws_big_lambda/big_lambdas_manager.sh sam_deploy qa
make deploy_qa
- Validate the SAM deployment templates on QA.
Runsmake create_s3_bucket_qa
,sh scripts/aws_big_lambda/big_lambdas_manager.sh sam_validate qa
make deploy_validate_qa
- Create the deployment QA package only.
Useful to check the package size and test the image by a local Docker run.
Runsmake create_s3_bucket_qa
,sh scripts/aws_big_lambda/big_lambdas_manager.sh package qa
.
make deploy_package_qa
- Deploy the App on Staging.
Runsmake create_s3_bucket_staging
,sh scripts/aws_big_lambda/big_lambdas_manager.sh sam_deploy staging
make deploy_staging
- Deploy the App on Production.
Runsmake create_s3_bucket_prod
,sh scripts/aws_big_lambda/big_lambdas_manager.sh sam_deploy prod
make deploy_prod
- Alias to run
make deploy_qa
.
make deploy
- Execute the App locally using the development database, asking to run it over
http
orhttps
.
Runsmake config_qa
,make clean_logs
, andsh scripts/aws/run_aws.sh run_local
. [???]
make run
- Execute the App locally using the QA database, asking to run it over
http
orhttps
.
Runsmake config_qa
,make clean_logs
, andsh scripts/aws/run_aws.sh run_local qa
.
make run_qa
- Runs
make config_qa
,make clean_logs
, andsh scripts/secure_local_server/run.sh "down" ""
Stop and destroy the App local Docker container (for any running stage).
make down_qa
- Restart the App local Docker container running over QA.
Runsmake config_qa
,make clean_logs
,sh scripts/secure_local_server/run.sh "down" ""
andsh scripts/aws/run_aws.sh run_local qa
.
make restart_qa
- Execute the App locally using the development database (in a local Docker container), asking to execute it over
http
orhttps
.
Runsmake config_local
,make clean_logs
,sh scripts/aws/run_aws.sh run_local dev
. [???]
make run_local_docker
- Execute
chalice local --port \$PORT --stage PROD
Runsmake config
,make clean_logs
,sh scripts/aws/run_aws.sh run
.
make run_prod
- Add the Git Submodule with the common JSON config directories.
Runssh scripts/add_github_submodules.sh
.
make add_submodules
- Start the local DNS Server.
Runssh scripts/dns/run_local_dns.sh
make local_dns
- Runs
sh scripts/dns/run_local_dns.sh restart
to restart the local DNS Server.
make local_dns_restart
- Restart and rebuild the local DNS Server configuration when the local IP or any DNS parameters has been changed.
Runssh scripts/dns/run_local_dns.sh rebuild
make local_dns_rebuild
- Stop and destroy the local DNS Server.
Runssh scripts/dns/run_local_dns.sh down
.
make local_dns_down
- Test the local DNS Server.
Runssh scripts/dns/run_local_dns.sh test
.
make local_dns_test
- Create the self-signed local SSL certificates (required to run the local development frontend and backend over https).
Runssh scripts/local_ssl_certs_creation.sh
.
make create_ssl_certs_only
- Copy the self-signed local SSL certificates to the frontend directory/local repository.
Runssh scripts/local_ssl_certs_copy.sh
.
make copy_ssl_certs
- Alias to run
make create_ssl_certs_only
andmake copy_ssl_certs
.
make create_ssl_certs
- Update the package.json file with the version and all other parameters except dependencies.
Runsnpm install --package-lock-only
.
make lock
- Test the publish to NPMJS without actually publishing.
Runssh scripts/npm_publish.sh pre-publish
.
make pre-publish
- Publish the scripts library to NPMJS.
Runssh scripts/npm_publish.sh publish
.
Requirements:
make publish
- Build 'dist' directory needed for the Pypi publish.
Runspoetry lock --no-update
,rm -rf dist
andpython3 -m build
.
Requirements:
make pypi-build
- Pypi Test publish.
Runsmake pypi-build
, andpython3 -m twine upload --repository testpypi dist/*
.
Requirements:
make pypi-publish-test
- Pypi Production publish
Runsmake pypi-build
, andpython3 -m twine upload dist/*
.
Requirements:
make pypi-publish
- If you get the error
Warning: Python >=3.9,<4.0 was not found on your system...
doingmake install
:
$ make install
pipenv install
Warning: Python >=3.9,<4.0 was not found on your system...
You can specify specific versions of Python with:
$ pipenv --python path/to/python
make: *** [install] Error 1
Fix it with these commands:
# Set the project Python version with pyenv
pyenv local 3.11
# Set the Python path with pipenv
pipenv --python ${HOME}/.pyenv/shims/python
And repeat make install
- If you get the warning
This version of npm is compatible with lockfileVersion@1...
doingmake install
:
npm install
npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@3. I'll try to do my best with it!
It's because you're using an old Node version. To solve it:
nvm node 18
And repeat make install
-
If you get
APP_NAME not set
message doing anymake run
, it's because the.env
file must be created or reviewed. Check the GenericsSuite (backend version) Configuration or GenericsSuite AI (backend version) Configuration -
If you get CORS errors in the frontend and backend communication:
- To make both use
localhost
andhttp
, change these variables in the.env
file:
- To make both use
# Frontend .env file:
APP_LOCAL_DOMAIN_NAME=localhost
# Backend .env file:
APP_CORS_ORIGIN_QA_LOCAL=http://localhost:3000
And make run
both frontend and backend with the http
option.
- To make both use the local DNS server and
https
, change these variables in the.env
file:
# Frontend .env file:
APP_LOCAL_DOMAIN_NAME=app.exampleapp.local
# Backend .env file:
APP_CORS_ORIGIN_QA_LOCAL=https://app.exampleapp.local:3000
NOTE: replace exampleapp
with your App name, all lowercased.
And make run
both frontend and backend with the https
option.
- If the local DNS Server seems to be unreachable or not working:
Restart the local backend server:
make local_dns_restart
If the local IP changes, make sure to:
- Run
make local_dns_rebuild
.
- Copy the
IP address
reported by the previous command.
E.g.
Local DNS domain 'app.exampleapp.local' is pointing to IP address '192.168.1.158'
.
- Run
make restart_qa
- Add the
IP address
to the DNS Servers in your computer'sNetwork > DNS servers
settings. The new DNS ServerIP address
must be the first one in the list of DNS servers.
- Restart the computer's WiFi or LAN network connection.
GenericSuite is open-sourced software licensed under the ISC license.
This project is developed and maintained by Carlos J. Ramirez. For more information or to contribute to the project, visit The GenericSuite Scripts (backend version) on GitHub.
Happy Coding!