Merge branch 'master' of https://github.com/simple-login/app into feature/custom_domain_random_suffix

This commit is contained in:
Sylvia van Os 2020-10-07 18:52:54 +02:00
commit 6b085960cb
33 changed files with 3191 additions and 364 deletions

View File

@ -22,15 +22,16 @@ jobs:
- uses: actions/cache@v1
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
path: ~/.cache/poetry
key: ${{ runner.os }}-poetry-${{ hashFiles('**/peotry.lock') }}
restore-keys: |
${{ runner.os }}-pip-
${{ runner.os }}-poetry-
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
python -m pip install poetry==1.0.10
poetry config virtualenvs.create false
poetry install
- name: Test formatting
run: |

View File

@ -4,16 +4,19 @@ WORKDIR /code
COPY ./static/package*.json /code/static/
RUN cd /code/static && npm install
# Main image
FROM python:3.7
WORKDIR /code
# install some utility packages
RUN apt update && apt install -y vim telnet
RUN pip3 install poetry==1.0.10
# install dependencies
COPY ./requirements.txt ./
RUN pip3 install --no-cache-dir -r requirements.txt
WORKDIR /code
COPY poetry.lock pyproject.toml ./
RUN poetry config virtualenvs.create false \
&& poetry install
# copy npm packages
COPY --from=npm /code /code

View File

@ -394,7 +394,7 @@ smtpd_recipient_restrictions =
```
Create the `/etc/postfix/pgsql-relay-domains.cf` file with the following content.
Make sure that the database config is correctly set and replace `mydomain.com` with your domain.
Make sure that the database config is correctly set, replace `mydomain.com` with your domain, update 'myuser' and 'mypassword' with your postgress credentials.
```
# postgres config
@ -408,7 +408,7 @@ query = SELECT domain FROM custom_domain WHERE domain='%s' AND verified=true
```
Create the `/etc/postfix/pgsql-transport-maps.cf` file with the following content.
Again, make sure that the database config is correctly set and replace `mydomain.com` with your domain.
Again, make sure that the database config is correctly set, replace `mydomain.com` with your domain, update 'myuser' and 'mypassword' with your postgress credentials.
```
# postgres config
@ -432,7 +432,7 @@ sudo systemctl restart postfix
To run the server, you need a config file. Please have a look at [config example](example.env) for an example to create one. Some parameters are optional and are commented out by default. Some have "dummy" values, fill them up if you want to enable these features (Paddle, AWS, etc).
Let's put your config file at `~/simplelogin.env`. Below is an example that you can use right away, make sure to replace `mydomain.com` by your domain and set `FLASK_SECRET` to a secret string.
Let's put your config file at `~/simplelogin.env`. Below is an example that you can use right away, make sure to replace `mydomain.com` by your domain, set `FLASK_SECRET` to a secret string, update 'myuser' and 'mypassword' with your postgress credentials.
Make sure to update the following variables and replace these values by yours.
@ -537,7 +537,7 @@ sudo docker run -d \
### Nginx
Install Nginx
Install Nginx and make sure to replace `mydomain.com` by your domain
```bash
sudo apt-get install -y nginx
@ -591,10 +591,15 @@ All work on SimpleLogin happens directly on GitHub.
### Run code locally
The project uses Python 3.7+ and Node v10. First, install all dependencies by running the following command. Feel free to use `virtualenv` or similar tools to isolate development environment.
The project uses
- Python 3.7+ and [poetry](https://python-poetry.org/) to manage dependencies
- Node v10 for front-end.
First, install all dependencies by running the following command.
Feel free to use `virtualenv` or similar tools to isolate development environment.
```bash
pip3 install -r requirements.txt
poetry install
```
You also need to install `gpg`, on Mac it can be done with:
@ -603,7 +608,6 @@ You also need to install `gpg`, on Mac it can be done with:
brew install gnupg
```
Then make sure all tests pass
```bash

View File

@ -57,6 +57,8 @@ def auth_login():
# Trigger rate limiter
g.deduct_limit = True
return jsonify(error="Email or password incorrect"), 400
elif user.disabled:
return jsonify(error="Account disabled"), 400
elif not user.activated:
return jsonify(error="Account not activated"), 400
elif user.fido_enabled():

View File

@ -36,6 +36,11 @@ def login():
g.deduct_limit = True
form.password.data = None
flash("Email or password incorrect", "error")
elif user.disabled:
flash(
"Your account is disabled. Please contact SimpleLogin team to re-enable your account.",
"error",
)
elif not user.activated:
show_resend_activation = True
flash(

View File

@ -178,7 +178,7 @@
title="Click to copy"
class="clipboard"
data-clipboard-text="{{ dkim_cname }}" style="overflow-wrap: break-word">
{{ dkim_cname }}
{{ dkim_cname }}.
</em>
</div>

View File

@ -193,6 +193,7 @@
>
<span class="font-weight-bold">{{ alias.email }}</span>
</span>
{% if alias.automatic_creation %} <span class="fa fa-inbox" data-toggle="tooltip" title="This alias was automatically generated because of an incoming email"></span>{% endif %}
</div>
<div class="col text-right">
<label class="custom-switch cursor"

View File

@ -315,10 +315,23 @@
<div class="mb-3">
You can download all aliases you have created on SimpleLogin along with other data.
</div>
<form method="post">
<input type="hidden" name="form-name" value="export-data">
<button class="btn btn-outline-info">Export Data</button>
</form>
<div class="d-flex">
<div>
<form method="post">
<input type="hidden" name="form-name" value="export-data">
<button class="btn btn-outline-info">Export Data</button>
</form>
</div>
<div class="ml-5">
<form method="post">
<input type="hidden" name="form-name" value="export-alias">
<button class="btn btn-outline-primary">Export Aliases</button>
</form>
</div>
</div>
</div>
</div>

View File

@ -1,8 +1,17 @@
import csv
import json
from io import BytesIO
from io import BytesIO, StringIO
import arrow
from flask import render_template, request, redirect, url_for, flash, Response
from flask import (
render_template,
request,
redirect,
url_for,
flash,
Response,
make_response,
)
from flask_login import login_required, current_user, logout_user
from flask_wtf import FlaskForm
from flask_wtf.file import FileField
@ -270,6 +279,18 @@ def setting():
mimetype="text/json",
headers={"Content-Disposition": "attachment;filename=data.json"},
)
elif request.form.get("form-name") == "export-alias":
data = [["alias", "note", "enabled"]]
for alias in Alias.filter_by(user_id=current_user.id).all(): # type: Alias
data.append([alias.email, alias.note, alias.enabled])
si = StringIO()
cw = csv.writer(si)
cw.writerows(data)
output = make_response(si.getvalue())
output.headers["Content-Disposition"] = "attachment; filename=aliases.csv"
output.headers["Content-type"] = "text/csv"
return output
manual_sub = ManualSubscription.get_by(user_id=current_user.id)
return render_template(

View File

@ -135,15 +135,6 @@ def send_change_email(new_email, current_email, name, link):
)
def send_new_app_email(email, name):
send_email(
email,
f"Any question/feedback for SimpleLogin {name}?",
render("com/new-app.txt", name=name),
render("com/new-app.html", name=name),
)
def send_test_email_alias(email, name):
send_email(
email,

View File

@ -162,6 +162,9 @@ class User(db.Model, ModelMixin, UserMixin):
activated = db.Column(db.Boolean, default=False, nullable=False)
# an account can be disabled if having harmful behavior
disabled = db.Column(db.Boolean, default=False, nullable=False, server_default="0")
profile_picture_id = db.Column(db.ForeignKey(File.id), nullable=True)
otp_secret = db.Column(db.String(16), nullable=True)
@ -1101,7 +1104,7 @@ class Contact(db.Model, ModelMixin):
pgp_public_key = db.Column(db.Text, nullable=True)
pgp_finger_print = db.Column(db.String(512), nullable=True)
alias = db.relationship(Alias)
alias = db.relationship(Alias, backref="contacts")
user = db.relationship(User)
# the latest reply sent to this contact
@ -1181,19 +1184,20 @@ class Contact(db.Model, ModelMixin):
or user.sender_format == SenderFormatEnum.VIA.value
):
new_name = f"{self.website_email} via SimpleLogin"
elif user.sender_format == SenderFormatEnum.AT.value:
name = self.name or ""
else:
if user.sender_format == SenderFormatEnum.AT.value:
formatted_email = self.website_email.replace("@", " at ").strip()
elif user.sender_format == SenderFormatEnum.A.value:
formatted_email = self.website_email.replace("@", "(a)").strip()
elif user.sender_format == SenderFormatEnum.FULL.value:
formatted_email = self.website_email.strip()
# Prefix name to formatted email if available
new_name = (
name + (" - " if name else "") + self.website_email.replace("@", " at ")
).strip()
elif user.sender_format == SenderFormatEnum.A.value:
name = self.name or ""
new_name = (
name + (" - " if name else "") + self.website_email.replace("@", "(a)")
).strip()
elif user.sender_format == SenderFormatEnum.FULL.value:
name = self.name or ""
new_name = (name + (" - " if name else "") + self.website_email).strip()
(self.name + " - " + formatted_email)
if self.name and self.name != self.website_email.strip()
else formatted_email
)
new_addr = formataddr((new_name, self.reply_email)).strip()
return new_addr.strip()
@ -1245,7 +1249,7 @@ class EmailLog(db.Model, ModelMixin):
refused_email = db.relationship("RefusedEmail")
forward = db.relationship(Contact)
contact = db.relationship(Contact)
contact = db.relationship(Contact, backref="email_logs")
def bounced_mailbox(self) -> str:
if self.bounced_mailbox_id:

133
app/spamassassin_utils.py Normal file
View File

@ -0,0 +1,133 @@
"""Inspired from
https://github.com/petermat/spamassassin_client
"""
import socket, select, re, logging
from io import BytesIO
from app.log import LOG
divider_pattern = re.compile(br"^(.*?)\r?\n(.*?)\r?\n\r?\n", re.DOTALL)
first_line_pattern = re.compile(br"^SPAMD/[^ ]+ 0 EX_OK$")
class SpamAssassin(object):
def __init__(self, message, timeout=20, host="127.0.0.1", spamd_user="spamd"):
self.score = None
self.symbols = None
self.spamd_user = spamd_user
# Connecting
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.settimeout(timeout)
client.connect((host, 783))
# Sending
client.sendall(self._build_message(message))
client.shutdown(socket.SHUT_WR)
# Reading
resfp = BytesIO()
while True:
ready = select.select([client], [], [], timeout)
if ready[0] is None:
# Kill with Timeout!
logging.info("[SpamAssassin] - Timeout ({0}s)!".format(str(timeout)))
break
data = client.recv(4096)
if data == b"":
break
resfp.write(data)
# Closing
client.close()
client = None
self._parse_response(resfp.getvalue())
def _build_message(self, message):
reqfp = BytesIO()
data_len = str(len(message)).encode()
reqfp.write(b"REPORT SPAMC/1.2\r\n")
reqfp.write(b"Content-Length: " + data_len + b"\r\n")
reqfp.write(f"User: {self.spamd_user}\r\n\r\n".encode())
reqfp.write(message)
return reqfp.getvalue()
def _parse_response(self, response):
if response == b"":
logging.info("[SPAM ASSASSIN] Empty response")
return None
match = divider_pattern.match(response)
if not match:
logging.error("[SPAM ASSASSIN] Response error:")
logging.error(response)
return None
first_line = match.group(1)
headers = match.group(2)
body = response[match.end(0) :]
# Checking response is good
match = first_line_pattern.match(first_line)
if not match:
logging.error("[SPAM ASSASSIN] invalid response:")
logging.error(first_line)
return None
report_list = [
s.strip() for s in body.decode("utf-8", errors="ignore").strip().split("\n")
]
linebreak_num = report_list.index([s for s in report_list if "---" in s][0])
tablelists = [s for s in report_list[linebreak_num + 1 :]]
self.report_fulltext = "\n".join(report_list)
# join line when current one is only wrap of previous
tablelists_temp = []
if tablelists:
for counter, tablelist in enumerate(tablelists):
if len(tablelist) > 1:
if (tablelist[0].isnumeric() or tablelist[0] == "-") and (
tablelist[1].isnumeric() or tablelist[1] == "."
):
tablelists_temp.append(tablelist)
else:
if tablelists_temp:
tablelists_temp[-1] += " " + tablelist
tablelists = tablelists_temp
# create final json
self.report_json = dict()
for tablelist in tablelists:
wordlist = re.split("\s+", tablelist)
try:
self.report_json[wordlist[1]] = {
"partscore": float(wordlist[0]),
"description": " ".join(wordlist[1:]),
}
except ValueError:
LOG.warning("Cannot parse %s %s", wordlist[0], wordlist)
headers = (
headers.decode("utf-8")
.replace(" ", "")
.replace(":", ";")
.replace("/", ";")
.split(";")
)
self.score = float(headers[2])
def get_report_json(self):
return self.report_json
def get_score(self):
return self.score
def is_spam(self, level=5):
return self.score is None or self.score > level
def get_fulltext(self):
return self.report_fulltext

View File

@ -49,6 +49,7 @@ import aiosmtpd
import aiospamc
import arrow
import spf
from aiosmtpd.controller import Controller
from aiosmtpd.smtp import Envelope
from sqlalchemy.exc import IntegrityError
@ -109,6 +110,7 @@ from app.models import (
Mailbox,
)
from app.pgp_utils import PGPException
from app.spamassassin_utils import SpamAssassin
from app.utils import random_string
from init_app import load_pgp_public_keys
from server import create_app, create_light_app
@ -436,9 +438,7 @@ def handle_email_sent_to_ourself(alias, mailbox, msg: Message, user):
)
async def handle_forward(
envelope, msg: Message, rcpt_to: str
) -> List[Tuple[bool, str]]:
def handle_forward(envelope, msg: Message, rcpt_to: str) -> List[Tuple[bool, str]]:
"""return an array of SMTP status (is_success, smtp_status)
is_success indicates whether an email has been delivered and
smtp_status is the SMTP Status ("250 Message accepted", "550 Non-existent email address", etc)
@ -453,6 +453,12 @@ async def handle_forward(
LOG.d("alias %s cannot be created on-the-fly, return 550", address)
return [(False, "550 SL E3 Email not exist")]
if alias.user.disabled:
LOG.exception(
"User %s disabled, disable forwarding emails for %s", alias.user, alias
)
return [(False, "550 SL E20 Account disabled")]
mail_from = envelope.mail_from
for mb in alias.mailboxes:
# email send from a mailbox to alias
@ -490,7 +496,7 @@ async def handle_forward(
return [(False, "550 SL E18 unverified mailbox")]
else:
ret.append(
await forward_email_to_mailbox(
forward_email_to_mailbox(
alias, msg, email_log, contact, envelope, mailbox, user
)
)
@ -502,7 +508,7 @@ async def handle_forward(
ret.append((False, "550 SL E19 unverified mailbox"))
else:
ret.append(
await forward_email_to_mailbox(
forward_email_to_mailbox(
alias,
copy(msg),
email_log,
@ -516,7 +522,7 @@ async def handle_forward(
return ret
async def forward_email_to_mailbox(
def forward_email_to_mailbox(
alias,
msg: Message,
email_log: EmailLog,
@ -566,7 +572,7 @@ async def forward_email_to_mailbox(
if SPAMASSASSIN_HOST:
start = time.time()
spam_score = await get_spam_score(msg)
spam_score = get_spam_score(msg)
LOG.d(
"%s -> %s - spam score %s in %s seconds",
contact,
@ -684,7 +690,7 @@ async def forward_email_to_mailbox(
return True, "250 Message accepted for delivery"
async def handle_reply(envelope, msg: Message, rcpt_to: str) -> (bool, str):
def handle_reply(envelope, msg: Message, rcpt_to: str) -> (bool, str):
"""
return whether an email has been delivered and
the smtp status ("250 Message accepted", "550 Non-existent email address", etc)
@ -713,6 +719,15 @@ async def handle_reply(envelope, msg: Message, rcpt_to: str) -> (bool, str):
user = alias.user
mail_from = envelope.mail_from
if user.disabled:
LOG.exception(
"User %s disabled, disable sending emails from %s to %s",
user,
alias,
contact,
)
return [(False, "550 SL E20 Account disabled")]
# bounce email initiated by Postfix
# can happen in case emails cannot be delivered to user-email
# in this case Postfix will try to send a bounce report to original sender, which is
@ -762,7 +777,7 @@ async def handle_reply(envelope, msg: Message, rcpt_to: str) -> (bool, str):
# do not use user.max_spam_score here
if SPAMASSASSIN_HOST:
start = time.time()
spam_score = await get_spam_score(msg)
spam_score = get_spam_score(msg)
LOG.d(
"%s -> %s - spam score %s in %s seconds",
alias,
@ -1418,7 +1433,7 @@ def handle_sender_email(envelope: Envelope):
return "250 email to sender accepted"
async def handle(envelope: Envelope) -> str:
def handle(envelope: Envelope) -> str:
"""Return SMTP status"""
# sanitize mail_from, rcpt_tos
@ -1455,7 +1470,7 @@ async def handle(envelope: Envelope) -> str:
# recipient starts with "reply+" or "ra+" (ra=reverse-alias) prefix
if rcpt_to.startswith("reply+") or rcpt_to.startswith("ra+"):
LOG.debug(">>> Reply phase %s(%s) -> %s", mail_from, msg["From"], rcpt_to)
is_delivered, smtp_status = await handle_reply(envelope, msg, rcpt_to)
is_delivered, smtp_status = handle_reply(envelope, msg, rcpt_to)
res.append((is_delivered, smtp_status))
else: # Forward case
LOG.debug(
@ -1464,9 +1479,7 @@ async def handle(envelope: Envelope) -> str:
msg["From"],
rcpt_to,
)
for is_delivered, smtp_status in await handle_forward(
envelope, msg, rcpt_to
):
for is_delivered, smtp_status in handle_forward(envelope, msg, rcpt_to):
res.append((is_delivered, smtp_status))
for (is_success, smtp_status) in res:
@ -1478,7 +1491,7 @@ async def handle(envelope: Envelope) -> str:
return res[0][1]
async def get_spam_score(message: Message) -> float:
async def get_spam_score_async(message: Message) -> float:
LOG.debug("get spam score for %s", message[_MESSAGE_ID])
sa_input = to_bytes(message)
@ -1502,6 +1515,25 @@ async def get_spam_score(message: Message) -> float:
return -999
def get_spam_score(message: Message) -> float:
LOG.debug("get spam score for %s", message[_MESSAGE_ID])
sa_input = to_bytes(message)
# Spamassassin requires to have an ending linebreak
if not sa_input.endswith(b"\n"):
LOG.d("add linebreak to spamassassin input")
sa_input += b"\n"
try:
# wait for at max 300s which is the default spamd timeout-child
sa = SpamAssassin(sa_input, host=SPAMASSASSIN_HOST, timeout=300)
return sa.get_score()
except Exception:
LOG.exception("SpamAssassin exception")
# return a negative score so the message is always considered as ham
return -999
def sl_sendmail(from_addr, to_addr, msg: Message, mail_options, rcpt_options):
"""replace smtp.sendmail"""
if POSTFIX_SUBMISSION_TLS:
@ -1522,12 +1554,9 @@ def sl_sendmail(from_addr, to_addr, msg: Message, mail_options, rcpt_options):
class MailHandler:
def __init__(self, lock):
self.lock = lock
async def handle_DATA(self, server, session, envelope: Envelope):
try:
ret = await self._handle(envelope)
ret = self._handle(envelope)
return ret
except Exception:
LOG.exception(
@ -1537,31 +1566,42 @@ class MailHandler:
)
return "421 SL Retry later"
async def _handle(self, envelope: Envelope):
async with self.lock:
start = time.time()
LOG.info(
"===>> New message, mail from %s, rctp tos %s ",
envelope.mail_from,
envelope.rcpt_tos,
)
def _handle(self, envelope: Envelope):
start = time.time()
LOG.info(
"===>> New message, mail from %s, rctp tos %s ",
envelope.mail_from,
envelope.rcpt_tos,
)
app = new_app()
with app.app_context():
ret = await handle(envelope)
LOG.info("takes %s seconds <<===", time.time() - start)
return ret
app = new_app()
with app.app_context():
ret = handle(envelope)
LOG.info("takes %s seconds <<===", time.time() - start)
return ret
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"-p", "--port", help="SMTP port to listen for", type=int, default=20381
)
args = parser.parse_args()
def main(port: int):
"""Use aiosmtpd Controller"""
controller = Controller(MailHandler(), hostname="0.0.0.0", port=port)
LOG.info("Listen for port %s", args.port)
controller.start()
LOG.d("Start mail controller %s %s", controller.hostname, controller.port)
if LOAD_PGP_EMAIL_HANDLER:
LOG.warning("LOAD PGP keys")
app = create_app()
with app.app_context():
load_pgp_public_keys()
while True:
time.sleep(2)
def asyncio_main(port: int):
"""
Main entrypoint using asyncio directly without passing by aiosmtpd Controller
"""
if LOAD_PGP_EMAIL_HANDLER:
LOG.warning("LOAD PGP keys")
app = create_app()
@ -1577,7 +1617,7 @@ if __name__ == "__main__":
return aiosmtpd.smtp.SMTP(handler, enable_SMTPUTF8=True)
server = loop.run_until_complete(
loop.create_server(factory, host="0.0.0.0", port=args.port)
loop.create_server(factory, host="0.0.0.0", port=port)
)
try:
@ -1590,3 +1630,14 @@ if __name__ == "__main__":
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"-p", "--port", help="SMTP port to listen for", type=int, default=20381
)
args = parser.parse_args()
LOG.info("Listen for port %s", args.port)
main(port=args.port)

View File

@ -58,7 +58,7 @@ def onboarding_send_from_alias(user):
send_email(
to_email,
f"Do you know you can send emails from your alias?",
f"SimpleLogin Tip: Send emails from your alias",
render("com/onboarding/send-from-alias.txt", user=user, to_email=to_email),
render("com/onboarding/send-from-alias.html", user=user, to_email=to_email),
)
@ -71,7 +71,7 @@ def onboarding_pgp(user):
send_email(
to_email,
f"Do you know you can encrypt your emails so only you can read them?",
f"SimpleLogin Tip: Secure your emails with PGP",
render("com/onboarding/pgp.txt", user=user, to_email=to_email),
render("com/onboarding/pgp.html", user=user, to_email=to_email),
)
@ -84,7 +84,7 @@ def onboarding_browser_extension(user):
send_email(
to_email,
f"Have you tried SimpleLogin Chrome/Firefox extensions and Android/iOS apps?",
f"SimpleLogin Tip: Chrome/Firefox/Safari extensions and Android/iOS apps",
render("com/onboarding/browser-extension.txt", user=user, to_email=to_email),
render("com/onboarding/browser-extension.html", user=user, to_email=to_email),
)
@ -97,7 +97,7 @@ def onboarding_mailbox(user):
send_email(
to_email,
f"Do you know you can have multiple mailboxes on SimpleLogin?",
f"SimpleLogin Tip: Multiple mailboxes",
render("com/onboarding/mailbox.txt", user=user, to_email=to_email),
render("com/onboarding/mailbox.html", user=user, to_email=to_email),
)

View File

@ -0,0 +1,29 @@
"""empty message
Revision ID: 1abfc9e14d7e
Revises: 58ad4df8583e
Create Date: 2020-10-04 12:47:43.738037
"""
import sqlalchemy_utils
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '1abfc9e14d7e'
down_revision = '58ad4df8583e'
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.add_column('users', sa.Column('disabled', sa.Boolean(), server_default='0', nullable=False))
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_column('users', 'disabled')
# ### end Alembic commands ###

View File

@ -8,11 +8,13 @@ from app.models import Monitoring
from server import create_app
# the number of consecutive fails
# if more than 3 fails, alert
# if more than _max_nb_fails, alert
# reset whenever the system comes back to normal
# a system is considered fail if incoming_queue + active_queue > 50
_nb_failed = 0
_max_nb_fails = 10
def get_stats():
"""Look at different metrics and alert appropriately"""
@ -35,7 +37,7 @@ def get_stats():
if incoming_queue + active_queue > 50:
_nb_failed += 1
if _nb_failed > 3:
if _nb_failed > _max_nb_fails:
# reset
_nb_failed = 0
@ -59,5 +61,5 @@ if __name__ == "__main__":
with app.app_context():
get_stats()
# 2 min
sleep(120)
# 1 min
sleep(60)

2700
poetry.lock generated Normal file

File diff suppressed because it is too large Load Diff

View File

@ -16,3 +16,69 @@ exclude = '''
)/
)
'''
[tool.poetry]
name = "SimpleLogin"
version = "0.1.0"
description = "open-source email alias solution"
authors = ["SimpleLogin <dev@simplelogin.io>"]
license = "MIT"
repository = "https://github.com/simple-login/app"
keywords = ["email", "alias", "privacy", "oauth2", "openid"]
[tool.poetry.dependencies]
python = "^3.7"
flask = "^1.1.2"
flask_sqlalchemy = "^2.4.4"
flask_login = "^0.5.0"
wtforms = "^2.3.3"
unidecode = "^1.1.1"
gunicorn = "^20.0.4"
bcrypt = "^3.2.0"
python-dotenv = "^0.14.0"
ipython = "^7.18.1"
sqlalchemy_utils = "^0.36.8"
psycopg2-binary = "^2.8.6"
sentry_sdk = "^0.18.0"
blinker = "^1.4"
arrow = "^0.16.0"
Flask-WTF = "^0.14.3"
boto3 = "^1.15.9"
Flask-Migrate = "^2.5.3"
flask_admin = "^1.5.6"
flask-cors = "^3.0.9"
watchtower = "^0.8.0"
sqlalchemy-utils = "^0.36.8"
jwcrypto = "^0.8"
yacron = "^0.11.1"
flask-debugtoolbar = "^0.11.0"
requests_oauthlib = "^1.3.0"
pyopenssl = "^19.1.0"
aiosmtpd = "^1.2"
dnspython = "^2.0.0"
coloredlogs = "^14.0"
pycryptodome = "^3.9.8"
phpserialize = "^1.3"
dkimpy = "^1.0.5"
pyotp = "^2.4.0"
flask_profiler = "^1.8.1"
facebook-sdk = "^3.1.0"
google-api-python-client = "^1.12.3"
google-auth-httplib2 = "^0.0.4"
python-gnupg = "^0.4.6"
webauthn = "^0.4.7"
pyspf = "^2.0.14"
Flask-Limiter = "^1.4"
memory_profiler = "^0.57.0"
gevent = "^20.9.0"
aiospamc = "^0.6.1"
email_validator = "^1.1.1"
[tool.poetry.dev-dependencies]
pytest = "^6.1.0"
black = "^20.8b1"
pre-commit = "^2.7.1"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

View File

@ -1,49 +0,0 @@
flask_sqlalchemy
flask
flask_login
wtforms
unidecode
gunicorn
pip-tools
bcrypt
python-dotenv
ipython
sqlalchemy_utils
psycopg2-binary
sentry_sdk
blinker
arrow
Flask-WTF
boto3
Flask-Migrate
flask_admin
pytest
flask-cors
watchtower
sqlalchemy-utils
jwcrypto
yacron
flask-debugtoolbar
requests_oauthlib
pyopenssl
aiosmtpd
dnspython
coloredlogs
pycryptodome
phpserialize
dkimpy
pyotp
flask_profiler
facebook-sdk
google-api-python-client
google-auth-httplib2
python-gnupg
webauthn
pyspf
Flask-Limiter
memory_profiler
gevent
aiocontextvars
aiospamc
black
pre-commit

View File

@ -1,147 +0,0 @@
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile
#
aiocontextvars==0.2.2 # via -r requirements.in
aiohttp==3.5.4 # via raven-aiohttp, yacron
aiosmtpd==1.2 # via -r requirements.in
aiosmtplib==1.0.6 # via yacron
aiospamc==0.6.1 # via -r requirements.in
alembic==1.0.10 # via flask-migrate
appdirs==1.4.4 # via black, virtualenv
appnope==0.1.0 # via ipython
arrow==0.14.2 # via -r requirements.in
asn1crypto==0.24.0 # via cryptography
async-timeout==3.0.1 # via aiohttp
atomicwrites==1.3.0 # via pytest
atpublic==1.0 # via aiosmtpd
attrs==19.1.0 # via aiohttp, pytest
backcall==0.1.0 # via ipython
bcrypt==3.1.6 # via -r requirements.in
black==20.8b1 # via -r requirements.in
blinker==1.4 # via -r requirements.in, flask-debugtoolbar
boto3==1.9.167 # via -r requirements.in, watchtower
botocore==1.12.167 # via boto3, s3transfer
cachetools==4.0.0 # via google-auth
cbor2==5.1.0 # via webauthn
certifi==2019.11.28 # via aiospamc, requests, sentry-sdk
cffi==1.12.3 # via bcrypt, cryptography
cfgv==3.2.0 # via pre-commit
chardet==3.0.4 # via aiohttp, requests
click==7.1.2 # via black, flask, pip-tools
coloredlogs==10.0 # via -r requirements.in
crontab==0.22.5 # via yacron
cryptography==2.7 # via jwcrypto, pyopenssl, webauthn
decorator==4.4.0 # via ipython, traitlets
distlib==0.3.1 # via virtualenv
dkimpy==1.0.1 # via -r requirements.in
dnspython==1.16.0 # via -r requirements.in, dkimpy
docutils==0.14 # via botocore
facebook-sdk==3.1.0 # via -r requirements.in
filelock==3.0.12 # via virtualenv
flask-admin==1.5.3 # via -r requirements.in
flask-cors==3.0.8 # via -r requirements.in
flask-debugtoolbar==0.10.1 # via -r requirements.in
flask-httpauth==3.3.0 # via flask-profiler
flask-limiter==1.3.1 # via -r requirements.in
flask-login==0.4.1 # via -r requirements.in
flask-migrate==2.5.2 # via -r requirements.in
flask-profiler==1.8.1 # via -r requirements.in
flask-sqlalchemy==2.4.0 # via -r requirements.in, flask-migrate
flask-wtf==0.14.2 # via -r requirements.in
flask==1.0.3 # via -r requirements.in, flask-admin, flask-cors, flask-debugtoolbar, flask-httpauth, flask-limiter, flask-login, flask-migrate, flask-profiler, flask-sqlalchemy, flask-wtf
future==0.18.2 # via webauthn
gevent==20.6.2 # via -r requirements.in
google-api-python-client==1.7.11 # via -r requirements.in
google-auth-httplib2==0.0.3 # via -r requirements.in, google-api-python-client
google-auth==1.11.2 # via google-api-python-client, google-auth-httplib2
greenlet==0.4.16 # via gevent
gunicorn==19.9.0 # via -r requirements.in
httplib2==0.17.0 # via google-api-python-client, google-auth-httplib2
humanfriendly==4.18 # via coloredlogs
identify==1.5.0 # via pre-commit
idna==2.8 # via requests, yarl
importlib-metadata==0.18 # via pluggy, pre-commit, pytest, virtualenv
ipython-genutils==0.2.0 # via traitlets
ipython==7.5.0 # via -r requirements.in
itsdangerous==1.1.0 # via flask, flask-debugtoolbar
jedi==0.13.3 # via ipython
jinja2==2.10.1 # via flask, yacron
jmespath==0.9.4 # via boto3, botocore
jwcrypto==0.6.0 # via -r requirements.in
limits==1.5.1 # via flask-limiter
mako==1.0.12 # via alembic
markupsafe==1.1.1 # via jinja2, mako
memory-profiler==0.57.0 # via -r requirements.in
more-itertools==7.0.0 # via pytest
multidict==4.5.2 # via aiohttp, yarl
mypy-extensions==0.4.3 # via black
nodeenv==1.5.0 # via pre-commit
oauthlib==3.0.2 # via requests-oauthlib
packaging==19.0 # via pytest
parso==0.4.0 # via jedi
pathspec==0.8.0 # via black
pexpect==4.7.0 # via ipython
phpserialize==1.3 # via -r requirements.in
pickleshare==0.7.5 # via ipython
pip-tools==5.3.1 # via -r requirements.in
pluggy==0.12.0 # via pytest
pre-commit==2.7.1 # via -r requirements.in
prompt-toolkit==2.0.9 # via ipython
psutil==5.7.0 # via memory-profiler
psycopg2-binary==2.8.2 # via -r requirements.in
ptyprocess==0.6.0 # via pexpect
py==1.8.0 # via pytest
pyasn1-modules==0.2.8 # via google-auth
pyasn1==0.4.8 # via pyasn1-modules, rsa
pycparser==2.19 # via cffi
pycryptodome==3.9.4 # via -r requirements.in
pygments==2.4.2 # via ipython
pyopenssl==19.0.0 # via -r requirements.in, webauthn
pyotp==2.3.0 # via -r requirements.in
pyparsing==2.4.0 # via packaging
pyspf==2.0.14 # via -r requirements.in
pytest==4.6.3 # via -r requirements.in
python-dateutil==2.8.0 # via alembic, arrow, botocore, strictyaml
python-dotenv==0.10.3 # via -r requirements.in
python-editor==1.0.4 # via alembic
python-gnupg==0.4.5 # via -r requirements.in
pyyaml==5.3.1 # via pre-commit
raven-aiohttp==0.7.0 # via yacron
raven==6.10.0 # via raven-aiohttp, yacron
regex==2020.7.14 # via black
requests-oauthlib==1.2.0 # via -r requirements.in
requests==2.22.0 # via facebook-sdk, requests-oauthlib
rsa==4.0 # via google-auth
ruamel.yaml==0.15.97 # via strictyaml
s3transfer==0.2.1 # via boto3
sentry-sdk==0.14.1 # via -r requirements.in
simplejson==3.17.0 # via flask-profiler
six==1.12.0 # via bcrypt, cryptography, flask-cors, flask-limiter, google-api-python-client, google-auth, limits, packaging, pip-tools, prompt-toolkit, pyopenssl, pytest, python-dateutil, sqlalchemy-utils, traitlets, virtualenv, webauthn
sqlalchemy-utils==0.36.1 # via -r requirements.in
sqlalchemy==1.3.19 # via alembic, flask-sqlalchemy, sqlalchemy-utils
strictyaml==1.0.2 # via yacron
toml==0.10.1 # via black, pre-commit
traitlets==4.3.2 # via ipython
typed-ast==1.4.1 # via black
typing-extensions==3.7.4.3 # via black
unidecode==1.0.23 # via -r requirements.in
uritemplate==3.0.1 # via google-api-python-client
urllib3==1.25.3 # via botocore, requests, sentry-sdk
virtualenv==20.0.31 # via pre-commit
watchtower==0.6.0 # via -r requirements.in
wcwidth==0.1.7 # via prompt-toolkit, pytest
webauthn==0.4.7 # via -r requirements.in
werkzeug==0.15.4 # via flask, flask-debugtoolbar
wtforms==2.2.1 # via -r requirements.in, flask-admin, flask-wtf
yacron==0.9.0 # via -r requirements.in
yarl==1.3.0 # via aiohttp
zipp==0.5.1 # via importlib-metadata
zope.event==4.4 # via gevent
zope.interface==5.1.0 # via gevent
# The following packages are considered to be unsafe in a requirements file:
# pip
# setuptools

View File

@ -41,7 +41,6 @@ from app.config import (
SENTRY_FRONT_END_DSN,
FIRST_ALIAS_DOMAIN,
SESSION_COOKIE_NAME,
ADMIN_EMAIL,
PLAUSIBLE_HOST,
PLAUSIBLE_DOMAIN,
GITHUB_CLIENT_ID,
@ -106,7 +105,7 @@ def create_light_app() -> Flask:
def create_app() -> Flask:
app = Flask(__name__)
# SimpleLogin is deployed behind NGINX
app.wsgi_app = ProxyFix(app.wsgi_app, num_proxies=1)
app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1, x_host=1)
limiter.init_app(app)
app.url_map.strict_slashes = False
@ -155,7 +154,7 @@ def create_app() -> Flask:
flask_profiler.init_app(app)
# enable CORS on /api endpoints
cors = CORS(app, resources={r"/api/*": {"origins": "*"}})
CORS(app, resources={r"/api/*": {"origins": "*"}})
# set session to permanent so user stays signed in after quitting the browser
# the cookie is valid for 7 days
@ -310,7 +309,9 @@ def fake_data():
@login_manager.user_loader
def load_user(user_id):
user = User.query.get(user_id)
user = User.get(user_id)
if user.disabled:
return None
return user
@ -657,14 +658,14 @@ window.location.href = "/";
"""
if __name__ == "__main__":
def local_main():
app = create_app()
# enable flask toolbar
app.config["DEBUG_TB_PROFILER_ENABLED"] = True
app.config["DEBUG_TB_INTERCEPT_REDIRECTS"] = False
app.debug = True
toolbar = DebugToolbarExtension(app)
DebugToolbarExtension(app)
# warning: only used in local
if RESET_DB:
@ -680,3 +681,7 @@ if __name__ == "__main__":
app.run(debug=True, port=7777, ssl_context=context)
else:
app.run(debug=True, port=7777)
if __name__ == "__main__":
local_main()

View File

@ -41,16 +41,6 @@ def reset_db():
create_db()
def send_safari_extension_newsletter():
for user in User.query.all():
send_email(
user.email,
"Quickly create alias with our Safari extension",
render("com/safari-extension.txt", user=user),
render("com/safari-extension.html", user=user),
)
def send_mailbox_newsletter():
for user in User.query.order_by(User.id).all():
if user.notification and user.activated:

60
static/package-lock.json generated vendored
View File

@ -5,59 +5,59 @@
"requires": true,
"dependencies": {
"@sentry/browser": {
"version": "5.21.4",
"resolved": "https://registry.npmjs.org/@sentry/browser/-/browser-5.21.4.tgz",
"integrity": "sha512-/bRGMNjJc4Qt9Me9qLobZe0pREUAMFQAR7GOF9HbgzxUc49qVvmPRglvwzwhPJ6XKPg0NH/C6MOn+yuIRjfMag==",
"version": "5.23.0",
"resolved": "https://registry.npmjs.org/@sentry/browser/-/browser-5.23.0.tgz",
"integrity": "sha512-lBBHb/NFDOy1K5E/noDkgaibTtxp8F8gmAaVhhpGvOjlcBp1wzNJhWRePYKWgjJ7yFudxGi4Qbferdhm9RwzbA==",
"requires": {
"@sentry/core": "5.21.4",
"@sentry/types": "5.21.4",
"@sentry/utils": "5.21.4",
"@sentry/core": "5.23.0",
"@sentry/types": "5.23.0",
"@sentry/utils": "5.23.0",
"tslib": "^1.9.3"
}
},
"@sentry/core": {
"version": "5.21.4",
"resolved": "https://registry.npmjs.org/@sentry/core/-/core-5.21.4.tgz",
"integrity": "sha512-2hB0shKL6RUuLqqmnDUPvwiV25OSnchxkJ6NbLqnn2DYLqLARfZuVcw2II4wb/Jlw7SDnbkQIPs0/ax7GPe1Nw==",
"version": "5.23.0",
"resolved": "https://registry.npmjs.org/@sentry/core/-/core-5.23.0.tgz",
"integrity": "sha512-K8Wp/g1opaauKJh2w5Z1Vw/YdudHQgH6Ng5fBazHZxA7zB9R8EbVKDsjy8XEcyHsWB7fTSlYX/7coqmZNOADdg==",
"requires": {
"@sentry/hub": "5.21.4",
"@sentry/minimal": "5.21.4",
"@sentry/types": "5.21.4",
"@sentry/utils": "5.21.4",
"@sentry/hub": "5.23.0",
"@sentry/minimal": "5.23.0",
"@sentry/types": "5.23.0",
"@sentry/utils": "5.23.0",
"tslib": "^1.9.3"
}
},
"@sentry/hub": {
"version": "5.21.4",
"resolved": "https://registry.npmjs.org/@sentry/hub/-/hub-5.21.4.tgz",
"integrity": "sha512-bgEgBHK6OWoAkrnYwVsIOw+sR4MWpe5/CB7H7r+GBJsSnBysncbSaBgndKmtb1GTWdzMxMlvXU16zC6TR5JX5Q==",
"version": "5.23.0",
"resolved": "https://registry.npmjs.org/@sentry/hub/-/hub-5.23.0.tgz",
"integrity": "sha512-P0sevLI9qAQc1J+AcHzNXwj83aG3GKiABVQJp0rgCUMtrXqLawa+j8pOHg8p7QWroHM7TKDMKeny9WemXBgzBQ==",
"requires": {
"@sentry/types": "5.21.4",
"@sentry/utils": "5.21.4",
"@sentry/types": "5.23.0",
"@sentry/utils": "5.23.0",
"tslib": "^1.9.3"
}
},
"@sentry/minimal": {
"version": "5.21.4",
"resolved": "https://registry.npmjs.org/@sentry/minimal/-/minimal-5.21.4.tgz",
"integrity": "sha512-pIpIH2ZTwdijGTw6VwfkTETAEoc9k/Aejz6mAjFDMzlOPb3bCx+W8EbGzFOxuwOsiE84bysd2UPVgFY4YSLV/g==",
"version": "5.23.0",
"resolved": "https://registry.npmjs.org/@sentry/minimal/-/minimal-5.23.0.tgz",
"integrity": "sha512-/w/B7ShMVu/tLI0/A5X+w6GfdZIQdFQihWyIK1vXaYS5NS6biGI3K6DcACuMrD/h4BsqlfgdXSOHHrmCJcyCXQ==",
"requires": {
"@sentry/hub": "5.21.4",
"@sentry/types": "5.21.4",
"@sentry/hub": "5.23.0",
"@sentry/types": "5.23.0",
"tslib": "^1.9.3"
}
},
"@sentry/types": {
"version": "5.21.4",
"resolved": "https://registry.npmjs.org/@sentry/types/-/types-5.21.4.tgz",
"integrity": "sha512-uJTRxW//NPO0UJJzRQOtYHg5tiSBvn1dRk5FvURXmeXt9d9XtwmRhHWDwI51uAkyv+51tun3v+0OZQfLvAI+gQ=="
"version": "5.23.0",
"resolved": "https://registry.npmjs.org/@sentry/types/-/types-5.23.0.tgz",
"integrity": "sha512-PbN5MVWxrq05sZ707lc8lleV0xSsI6jWr9h9snvbAuMjcauE0lmdWmjoWKY3PAz2s1mGYFh55kIo8SmQuVwbYg=="
},
"@sentry/utils": {
"version": "5.21.4",
"resolved": "https://registry.npmjs.org/@sentry/utils/-/utils-5.21.4.tgz",
"integrity": "sha512-zY8OvaE/lU+DCzTSFrDZNXZmBLM/0URUlyYD4RubqzrgKY/eP1pSbEsDzYYhc+OrBr8TjG66N+5T3gMZX0BfNg==",
"version": "5.23.0",
"resolved": "https://registry.npmjs.org/@sentry/utils/-/utils-5.23.0.tgz",
"integrity": "sha512-D5gQDM0wEjKxhE+YNvCuCHo/6JuaORF2/3aOhoJBR+dy9EACRspg7kp3+9KF44xd2HVEXkSVCJkv8/+sHePYRQ==",
"requires": {
"@sentry/types": "5.21.4",
"@sentry/types": "5.23.0",
"tslib": "^1.9.3"
}
},

2
static/package.json vendored
View File

@ -16,7 +16,7 @@
},
"homepage": "https://github.com/simple-login/app#readme",
"dependencies": {
"@sentry/browser": "^5.21.4",
"@sentry/browser": "^5.23.0",
"bootbox": "^5.4.0",
"font-awesome": "^4.7.0",
"intro.js": "^2.9.3",

View File

@ -1,11 +0,0 @@
{% extends "base.html" %}
{% block content %}
{{ render_text("Hi " + name) }}
{{ render_text("This is Son, SimpleLogin founder.") }}
{{ render_text("Even though I lead the company, Im the *product person* and the user experience you get from our product means a lot to me.") }}
{{ render_text('Our users and developers love SimpleLogin and its simplicity (hence the "simple" in the name), but if there\'s anything that\'s bugging you, even the smallest of issues that could be done better, I want to hear about it - so hit the reply button.') }}
{{ render_button("SimpleLogin documentation", "https://docs.simplelogin.io") }}
{{ render_text('Thanks, <br />SimpleLogin Team.') }}
{% endblock %}

View File

@ -1,10 +0,0 @@
Hi {{name}}
This is Son, SimpleLogin Founder 😊.
Even though I lead the company, Im the "product person" and the user experience you get from our product means a lot to me.
Our users and developers love SimpleLogin and its simplicity (hence the "simple" in the name 😉), but if there's anything that's bugging you, even the smallest of issues that could be done better, I want to hear about it - so hit the reply button.
Thanks!
SimpleLogin Team.

View File

@ -1,7 +1,11 @@
{% extends "base.html" %}
{% block content %}
{{ render_text("Hi " + user.name) }}
{% call text() %}
<h1>
Download SimpleLogin browser extensions and mobile apps to create aliases on-the-fly.
</h1>
{% endcall %}
{% call text() %}
If you want to quickly create aliases <b>without</b> going to SimpleLogin website, you can do that with SimpleLogin

View File

@ -1,7 +1,11 @@
{% extends "base.html" %}
{% block content %}
{{ render_text("Hi " + user.name) }}
{% call text() %}
<h1>
Add other mailboxes to SimpleLogin.
</h1>
{% endcall %}
{% call text() %}
If you have several email addresses, e.g. Gmail for work and Protonmail for personal uses,

View File

@ -1,7 +1,11 @@
{% extends "base.html" %}
{% block content %}
{{ render_text("Hi " + user.name) }}
{% call text() %}
<h1>
Secure your emails with PGP.
</h1>
{% endcall %}
{% call text() %}
If you use Gmail, Yahoo, Outlook, etc, you might want to use

View File

@ -1,10 +1,10 @@
{% extends "base.html" %}
{% block content %}
{{ render_text("Hi " + user.name) }}
{% call text() %}
Do you know you can send emails <b>from your alias</b>? <br>
<h1>
Send emails from your alias.
</h1>
{% endcall %}
{% call text() %}

View File

@ -133,8 +133,19 @@ def test_new_addr(flask_client):
)
assert c1.new_addr() == '"abcd@example.com via SimpleLogin" <rep@SL>'
# set sender format = FULL
user.sender_format = SenderFormatEnum.FULL.value
db.session.commit()
assert c1.new_addr() == '"First Last - abcd@example.com" <rep@SL>'
# Make sure email isn't duplicated if sender name equals email
c1.name = "abcd@example.com"
db.session.commit()
assert c1.new_addr() == '"abcd@example.com" <rep@SL>'
# set sender_format = AT
user.sender_format = SenderFormatEnum.AT.value
c1.name = "First Last"
db.session.commit()
assert c1.new_addr() == '"First Last - abcd at example.com" <rep@SL>'