diff --git a/.env.example b/.env.example index ac76ee16..6b060098 100644 --- a/.env.example +++ b/.env.example @@ -86,3 +86,7 @@ GOOGLE_CLIENT_SECRET=to_fill # Facebook FACEBOOK_CLIENT_ID=to_fill FACEBOOK_CLIENT_SECRET=to_fill + +# Flask profiler +# FLASK_PROFILER_PATH=/tmp/flask-profiler.sql +# FLASK_PROFILER_PASSWORD=password \ No newline at end of file diff --git a/.github/workflows/pythonpackage.yml b/.github/workflows/pythonpackage.yml index a76a912a..7cbc47a2 100644 --- a/.github/workflows/pythonpackage.yml +++ b/.github/workflows/pythonpackage.yml @@ -9,7 +9,7 @@ jobs: strategy: max-parallel: 4 matrix: - python-version: [3.7] + python-version: [3.6, 3.7] steps: - uses: actions/checkout@v1 diff --git a/Dockerfile b/Dockerfile index 110ead14..464787a6 100644 --- a/Dockerfile +++ b/Dockerfile @@ -13,5 +13,5 @@ RUN pip3 install -r requirements.txt COPY . . #gunicorn wsgi:app -b 0.0.0.0:7777 -w 2 --timeout 15 --log-level DEBUG -CMD ["gunicorn","wsgi:app","-b","0.0.0.0:7777","-w","2","--timeout","15","--log-level","DEBUG"] +CMD ["gunicorn","wsgi:app","-b","0.0.0.0:7777","-w","2","--timeout","15"] diff --git a/README.md b/README.md index 4e8f640d..66e46207 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,70 @@ -SimpleLogin - privacy-first email alias and Single Sign-On (SSO) Identity Provider ---- +
+ + + +
-https://simplelogin.io +[SimpleLogin](https://simplelogin.io) | Privacy-First Email Forwarding and Identity Provider Service +--- ++ + + + + + + + + + + + + + + + + + + + +
> Yet another email forwarding service? In some way yes... However, SimpleLogin is a bit different because: -- it's fully open-source: both the server and client code (browser extension, JS library) are open-source so anyone can freely inspect and (hopefully) improve the code. -- not just email alias: SimpleLogin is a privacy-first and developer-friendly identity provider that: a. offers privacy for users b. is simple to use for developers. Our goal is to offer a privacy-focused alternative to the "Login with Facebook/Google/Twitter" buttons. -- the only email alias solution that is `self-hostable`: with our detailed self-hosting instructions and most of components running as Docker container, anyone who knows how to `ssh` is able to deploy SimpleLogin on their server. -- plenty of features: custom domain, browser extension, alias activity, OAuth libraries, etc. -- written in Python 🐍 😅 this is not a difference per se but hey I never found a Python email server so feel free to tweak this one if you want to use Python for handling emails. +- Fully open source: both the server and client code (browser extension, JS library) are open source so anyone can freely inspect and (hopefully) improve the code. +- Not just email alias: SimpleLogin is a privacy-first and developer-friendly identity provider that: + - offers privacy for users + - is simple to use for developers. SimpleLogin is a privacy-focused alternative to the "Login with Facebook/Google/Twitter" buttons. + +- The only email forwarding solution that is **self-hostable**: with our detailed self-hosting instructions and most of components running as Docker container, anyone who knows `ssh` is able to deploy SimpleLogin on their server. +- Plenty of features: browser extension, custom domain, catch-all alias, OAuth libraries, etc. +- Open roadmap at https://trello.com/b/4d6A69I4/open-roadmap: you know the exciting features we are working on. + +At the heart of SimpleLogin is `email alias`: an alias is a normal email address but all emails sent to an alias are **forwarded** to your email inbox. SimpleLogin alias can also **send** emails: for your contact, the alias is therefore your email address. Use alias whenever you need to give out your email address to protect your online identity. More info on our website at https://simplelogin.io + ++ +
+ +# Quick start + +If you have Docker installed, run the following command to start SimpleLogin local server: + + +```bash +docker run -it --rm \ + -e RESET_DB=true \ + -e CONFIG=/code/.env.example \ + -p 7777:7777 \ + simplelogin/app python server.py +``` + +Then open http://localhost:7777, you should be able to login with `john@wick.com/password` account! + +To use SimpleLogin aliases, you need to deploy it on your server with some DNS setup though, +the following section will show a step-by-step guide on how to get your own email forwarder service! # Table of Contents @@ -24,13 +77,15 @@ In some way yes... However, SimpleLogin is a bit different because: ## General Architecture -![](docs/archi.png) ++ +
-SimpleLogin backend consists of 2 main components: +SimpleLogin backend consists of 2 main components: - the `webapp` used by several clients: web UI (the dashboard), browser extension (Chrome & Firefox for now), OAuth clients (apps that integrate "Login with SimpleLogin" button) and mobile app (work in progress). -- the `email handler`: implements the email forwarding (i.e. alias receiving email) and email sending (i.e. alias sending email). +- the `email handler`: implements the email forwarding (i.e. alias receiving email) and email sending (i.e. alias sending email). ## Self hosting @@ -40,11 +95,10 @@ SimpleLogin backend consists of 2 main components: - a domain that you can config the DNS. It could be a sub-domain. In the rest of the doc, let's say it's `mydomain.com` for the email and `app.mydomain.com` for SimpleLogin webapp. Please make sure to replace these values by your domain name whenever they appear in the doc. -- [Optional]: a dedicated Postgres database. If you don't want to manage and maintain a Postgres database, you can use managed services proposed by some cloud providers. Otherwise this guide will show how to run a Postgres database using Docker. Database is not well-known to be run inside Docker but this is probably fine if you don't have thousands of email addresses. - - [Optional] AWS S3, Sentry, Google/Facebook/Github developer accounts. These are necessary only if you want to activate these options. -All the below steps, except for the DNS ones that are usually done inside your domain registrar interface, are done on your server. The commands are to run with `bash` (or any bash-compatible shell like `zsh`) being the shell. If you use other shells like `fish`, please make sure to adapt the commands. + +Except for the DNS setup that is usually done on your domain registrar interface, all the below steps are to be done on your server. The commands are to run with `bash` (or any bash-compatible shell like `zsh`) being the shell. If you use other shells like `fish`, please make sure to adapt the commands. ### DKIM @@ -72,7 +126,7 @@ Please note that DNS changes could take up to 24 hours to propagate. In practice #### MX record Create a **MX record** that points `mydomain.com` to `app.mydomain.com` with priority 10. -To verify if the DNS works, `dig mydomain.com mx` should contain the following in the result. +To verify if the DNS works, `dig mydomain.com mx` should contain the following in the result. ``` mydomain.com. 3600 IN MX 10 app.mydomain.com. @@ -92,7 +146,7 @@ with `PUBLIC_KEY` being your `dkim.pub.key` but - remove the `-----BEGIN PUBLIC KEY-----` and `-----END PUBLIC KEY-----` - join all the lines on a single line. -For example, if your `dkim.pub.key` is +For example, if your `dkim.pub.key` is ``` -----BEGIN PUBLIC KEY----- @@ -113,16 +167,14 @@ From Wikipedia https://en.wikipedia.org/wiki/Sender_Policy_Framework > Sender Policy Framework (SPF) is an email authentication method designed to detect forging sender addresses during the delivery of the email -Similar to DKIM, setting up SPF is highly recommended. +Similar to DKIM, setting up SPF is highly recommended. Add a TXT record for `mydomain.com` with the value `v=spf1 mx -all`. What it means is only your server can send email with `@mydomain.com` domain. To verify, you can use `dig mydomain.com txt` -#### DMARC (optional) TODO - ### Docker Now the boring DNS stuffs are done, let's do something more fun! -Please follow the steps on [Docker CE for Ubuntu](https://docs.docker.com/v17.12/install/linux/docker-ce/ubuntu/) to install Docker on the server. +If you don't already have Docker installed on your server, please follow the steps on [Docker CE for Ubuntu](https://docs.docker.com/v17.12/install/linux/docker-ce/ubuntu/) to install Docker. Tips: if you want to run Docker without the `sudo` prefix, add your account to `docker` group: @@ -144,7 +196,7 @@ docker network create -d bridge \ ### Postgres -This section shows how to run a Postgres database using Docker. At the end of this section, you will have a database username and password which are being referred to the next steps. +This section shows how to run a Postgres database using Docker. At the end of this section, you will have a database username and password which will be used in the next steps. If you have already had a Postgres database in use, you can skip this section and just copy the database configuration (i.e. host, port, username, password, database name). @@ -176,7 +228,7 @@ Install `postfix` and `postfix-pgsql`. The latter is used to connect Postfix and sudo apt-get install -y postfix postfix-pgsql ``` -Choose "Internet Site" in Postfix installation window then keep using the proposed value as *System mail name* in the next window. +Choose "Internet Site" in Postfix installation window then keep using the proposed value as *System mail name* in the next window. Run the following commands to setup Postfix. Make sure to replace `mydomain.com` with the appropriate value of your domain. @@ -224,7 +276,7 @@ Finally, restart Postfix ### Run SimpleLogin Docker containers -To run the server, you need a config file. Please have a look at [config example](./.env.example) for an example to create one. Some parameters are optional and are commented out by default. Some have "dummy" values, fill them up if you want to enable these features (Paddle, AWS). +To run the server, you need a config file. Please have a look at [config example](./.env.example) for an example to create one. Some parameters are optional and are commented out by default. Some have "dummy" values, fill them up if you want to enable these features (Paddle, AWS, etc). Let's put your config file at `~/simplelogin.env`. @@ -239,9 +291,6 @@ EMAIL_SERVERS_WITH_PRIORITY=[(10, "app.mydomain.com.")] DKIM_PRIVATE_KEY_PATH=/dkim.key DKIM_PUBLIC_KEY_PATH=/dkim.pub.key DB_URI=postgresql://myuser:mypassword@sl-db:5432/simplelogin - -# optional, to have more choices for random alias. -WORDS_FILE_PATH=local_data/words_alpha.txt ``` @@ -327,11 +376,11 @@ At this step, you should also setup the SSL for Nginx. [Certbot](https://certbot ### Enjoy! -If all of the above steps are successful, open http://app.mydomain.com/ and create your first account! +If all of the above steps are successful, open http://app.mydomain.com/ and create your first account! ## Contributing -All work on SimpleLogin happens directly on GitHub. +All work on SimpleLogin happens directly on GitHub. ### Run code locally @@ -372,56 +421,120 @@ john@wick.com / password ### API -For now the only API client is the Chrome/Firefox extension. This extension relies on `API Code` for authentication. +For now the only API client is the Chrome/Firefox extension. This extension relies on `API Code` for authentication. In every request, the extension sends - the `API Code` is set in `Authentication` header. The check is done via the `verify_api_key` wrapper, implemented in `app/api/base.py` -- the current website `hostname` which is the website subdomain name + domain name. For ex, if user is on `http://dashboard.example.com/path1/path2?query`, the subdomain is `dashboard.example.com`. This information is important to know where an alias is used in order to proposer to user the same alias if they want to create on alias on the same website in the future. The `hostname` is passed in the request query `?hostname=`, see `app/api/views/alias_options.py` for an example. +- (Optional but recommended) `hostname` passed in query string. hostname is the the URL hostname (cf https://en.wikipedia.org/wiki/URL), for ex if URL is http://www.example.com/index.html then the hostname is `www.example.com`. This information is important to know where an alias is used in order to suggest user the same alias if they want to create on alias on the same website in the future. -Currently, the latest extension uses the two following endpoints : +If error, the API returns 4** with body containing the error message, for example: -- `/alias/options`: returns what to suggest to user when they open the extension. - -``` -GET /alias/options hostname?="www.groupon.com" - -Response: a json with following structure. ? means optional field. - recommendation?: - alias: www_groupon_com@simplelogin.co - hostname: www.groupon.com - - custom: - suggestion: groupon - suffix: [@my_domain.com, .abcde@simplelogin.co] - - can_create_custom: true - - existing: - [email1, email2, ...] +```json +{ + "error": "request body cannot be empty" +} ``` -- `/alias/custom/new`: allows user to create a new custom alias. +The error message could be displayed to user as-is, for example for when user exceeds their alias quota. +Some errors should be fixed during development however: for example error like `request body cannot be empty` is there to catch development error and should never be shown to user. + +All following endpoint return `401` status code if the API Key is incorrect. -To try out the endpoint, you can use the following command. The command uses [httpie](https://httpie.org). -Make sure to replace `{api_key}` by your API Key obtained on https://app.simplelogin.io/dashboard/api_key +#### GET /api/user_info -``` -http https://app.simplelogin.io/api/alias/options \ - Authentication:{api_key} \ - hostname==www.google.com +Given the API Key, return user name and whether user is premium. +This endpoint could be used to validate the api key. + +Input: +- `Authentication` header that contains the api key + +Output: if api key is correct, return a json with user name and whether user is premium, for example: + +```json +{ + "name": "John Wick", + "is_premium": false +} ``` +If api key is incorrect, return 401. + + +#### GET /api/v2/alias/options + +User alias info and suggestion. Used by the first extension screen when user opens the extension. + +Input: +- `Authentication` header that contains the api key +- (Optional but recommended) `hostname` passed in query string. + +Output: a json with the following field: +- can_create: boolean. Whether user can create new alias +- suffixes: list of string. List of alias `suffix` that user can use. If user doesn't have custom domain, this list has a single element which is the alias default domain (simplelogin.co). +- prefix_suggestion: string. Suggestion for the `alias prefix`. Usually this is the website name extracted from `hostname`. If no `hostname`, then the `prefix_suggestion` is empty. +- existing: list of string. List of existing alias. +- recommendation: optional field, dictionary. If an alias is already used for this website, the recommendation will be returned. There are 2 subfields in `recommendation`: `alias` which is the recommended alias and `hostname` is the website on which this alias is used before. + +For ex: +```json +{ + "can_create": true, + "existing": [ + "my-first-alias.meo@sl.local", + "e1.cat@sl.local", + "e2.chat@sl.local", + "e3.cat@sl.local" + ], + "prefix_suggestion": "test", + "recommendation": { + "alias": "e1.cat@sl.local", + "hostname": "www.test.com" + }, + "suffixes": [ + "@very-long-domain.com.net.org", + "@ab.cd", + ".cat@sl.local" + ] +} ``` -POST /alias/custom/new - prefix: www_groupon_com - suffix: @my_domain.com -Response: - 201 -> OK {alias: "www_groupon_com@my_domain.com"} - 409 -> duplicated +#### POST /api/alias/custom/new +Create a new custom alias. + +Input: +- `Authentication` header that contains the api key +- (Optional but recommended) `hostname` passed in query string +- Request Message Body in json (`Content-Type` is `application/json`) + - alias_prefix: string. The first part of the alias that user can choose. + - alias_suffix: should be one of the suffixes returned in the `GET /api/v2/alias/options` endpoint. + +Output: +If success, 201 with the new alias, for example + +```json +{ + "alias": "www_groupon_com@my_domain.com" +} +``` + +#### POST /api/alias/random/new + +Create a new random alias. + +Input: +- `Authentication` header that contains the api key +- (Optional but recommended) `hostname` passed in query string + +Output: +If success, 201 with the new alias, for example + +```json +{ + "alias": "www_groupon_com@my_domain.com" +} ``` ### Database migration @@ -430,7 +543,7 @@ The database migration is handled by `alembic` Whenever the model changes, a new migration has to be created -Set the database connection to use a current database (i.e. the one without the model changes you just made), for example, if you have a staging config at `~/config/simplelogin/staging.env`, you can do: +Set the database connection to use a current database (i.e. the one without the model changes you just made), for example, if you have a staging config at `~/config/simplelogin/staging.env`, you can do: ```bash ln -sf ~/config/simplelogin/staging.env .env @@ -448,16 +561,16 @@ In local the database creation in Sqlite doesn't use migration and uses directly The repo consists of the three following entry points: -- wsgi.py and server.py: the webapp. -- email_handler.py: the email handler. -- cron.py: the cronjob. +- wsgi.py and server.py: the webapp. +- email_handler.py: the email handler. +- cron.py: the cronjob. Here are the small sum-ups of the directory structures and their roles: - app/: main Flask app. It is structured into different packages representing different features like oauth, api, dashboard, etc. - local_data/: contains files to facilitate the local development. They are replaced during the deployment. - migrations/: generated by flask-migrate. Edit these files will be only edited when you spot (very rare) errors on the database migration files. -- static/: files available at `/static` url. +- static/: files available at `/static` url. - templates/: contains both html and email templates. - tests/: tests. We don't really distinguish unit, functional or integration test. A test is simply here to make sure a feature works correctly. @@ -473,7 +586,7 @@ SL currently supports code and implicit flow. #### Code flow -To trigger the code flow locally, you can go to the following url after running `python server.py`: +To trigger the code flow locally, you can go to the following url after running `python server.py`: ``` http://localhost:7777/oauth/authorize?client_id=client-id&state=123456&response_type=code&redirect_uri=http%3A%2F%2Flocalhost%3A7000%2Fcallback&state=random_string @@ -495,8 +608,8 @@ http http://localhost:7777/oauth/user_info 'Authorization:Bearer {token}' #### Implicit flow -Similar to code flow, except for the the `access token` which we we get back with the redirection. -For implicit flow, the url is +Similar to code flow, except for the the `access token` which we we get back with the redirection. +For implicit flow, the url is ``` http://localhost:7777/oauth/authorize?client_id=client-id&state=123456&response_type=token&redirect_uri=http%3A%2F%2Flocalhost%3A7000%2Fcallback&state=random_string @@ -533,6 +646,16 @@ response_type=id_token code ``` +## ❤️ Contributors +Thanks go to these wonderful people: - +Dung Nguyen Van |
+ Giuseppe Federico |
+ Ninh Dinh |
+ Tung Nguyen V. N. |
+ Son Nguyen Kim |
+
- To send an email from your alias, just send the email to a special email address that we call reverse-alias - and SimpleLogin will send it from the alias. + To send an email from your alias, just send the email to a special email address that we call + reverse-alias + and SimpleLogin will send it from the alias to the destination.
Make sure you send the email from your personal email address ({{ current_user.email }}). @@ -37,23 +38,25 @@