# HG changeset patch # User Brian Neal # Date 1442971430 18000 # Node ID 4f265f61874be458914340dd5c38906c5ed9948c # Parent bd594bcba5eb3f3879b2fb20682e9557fcec92e3 Hotlink image form is functioning. The user can now submit a URL via a form and the URL will be downloaded and uploaded to a S3 bucket if it is an image. Tests to follow. diff -r bd594bcba5eb -r 4f265f61874b bio/forms.py --- a/bio/forms.py Sun Sep 13 14:51:33 2015 -0500 +++ b/bio/forms.py Tue Sep 22 20:23:50 2015 -0500 @@ -15,7 +15,7 @@ from bio.models import UserProfile from core.widgets import AutoCompleteUserInput -from core.image import parse_image, downscale_image_square +from core.images.utils import parse_image, downscale_image_square class EditUserForm(forms.ModelForm): diff -r bd594bcba5eb -r 4f265f61874b core/image.py --- a/core/image.py Sun Sep 13 14:51:33 2015 -0500 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,118 +0,0 @@ -""" -This file contains common utility functions for manipulating images for -the rest of the applications in the project. -""" -from PIL import ImageFile -from PIL import Image - - -def parse_image(file): - """ - Returns a PIL Image from the supplied Django file object. - Throws IOError if the file does not parse as an image file or some other - I/O error occurred. - - """ - parser = ImageFile.Parser() - for chunk in file.chunks(): - parser.feed(chunk) - image = parser.close() - return image - - -def downscale_image_square(image, size): - """ - Scale an image to the square dimensions given by size (in pixels). - The new image is returned. - If the image is already smaller than (size, size) then no scaling - is performed and the image is returned unchanged. - - """ - # don't upscale - if (size, size) >= image.size: - return image - - (w, h) = image.size - if w > h: - diff = (w - h) / 2 - image = image.crop((diff, 0, w - diff, h)) - elif h > w: - diff = (h - w) / 2 - image = image.crop((0, diff, w, h - diff)) - image = image.resize((size, size), Image.ANTIALIAS) - return image - - -# Various image transformation functions: -def flip_horizontal(im): - return im.transpose(Image.FLIP_LEFT_RIGHT) - -def flip_vertical(im): - return im.transpose(Image.FLIP_TOP_BOTTOM) - -def rotate_180(im): - return im.transpose(Image.ROTATE_180) - -def rotate_90(im): - return im.transpose(Image.ROTATE_90) - -def rotate_270(im): - return im.transpose(Image.ROTATE_270) - -def transpose(im): - return rotate_90(flip_horizontal(im)) - -def transverse(im): - return rotate_90(flip_vertical(im)) - -# From http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/EXIF.html -# EXIF Orientation tag values: -# 1 = Horizontal (normal) -# 2 = Mirror horizontal -# 3 = Rotate 180 -# 4 = Mirror vertical -# 5 = Mirror horizontal and rotate 270 CW -# 6 = Rotate 90 CW -# 7 = Mirror horizontal and rotate 90 CW -# 8 = Rotate 270 CW - -ORIENT_FUNCS = { - 2: flip_horizontal, - 3: rotate_180, - 4: flip_vertical, - 5: transpose, - 6: rotate_270, - 7: transverse, - 8: rotate_90, -} - -ORIENT_TAG = 0x112 - - -def orient_image(im): - """Transforms the given image according to embedded EXIF data. - - The image instance, im, should be a PIL Image. - If there is EXIF information for the image, and the orientation tag - indicates that the image should be transformed, perform the transformation. - - Returns a tuple of the form (flag, image) where flag is True if the image - was oriented and False otherwise. image is either a new transformed image or - the original image instance. - - """ - if hasattr(im, '_getexif'): - try: - exif = im._getexif() - except IndexError: - # Work around issue seen in Pillow - # https://github.com/python-pillow/Pillow/issues/518 - exif = None - - if exif and ORIENT_TAG in exif: - orientation = exif[ORIENT_TAG] - func = ORIENT_FUNCS.get(orientation) - if func: - return (True, func(im)) - - return (False, im) diff -r bd594bcba5eb -r 4f265f61874b core/image_uploader.py --- a/core/image_uploader.py Sun Sep 13 14:51:33 2015 -0500 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,94 +0,0 @@ -"""This module contains a function to upload an image file to a S3Bucket. - -The image can be resized and a thumbnail can be generated and uploaded as well. - -""" -from base64 import b64encode -import logging -from io import BytesIO -import os.path -import uuid - -from PIL import Image - -from core.image import orient_image - - -logger = logging.getLogger(__name__) - - -def make_key(): - """Generate a random key suitable for a filename""" - return b64encode(uuid.uuid4().bytes, '-_').rstrip('=') - - -def upload(filename, bucket, metadata=None, new_size=None, thumb_size=None): - """Upload an image file to a given S3Bucket. - - The image can optionally be resized and a thumbnail can be generated and - uploaded as well. - - Parameters: - filename - The path to the file to process. The filename should have an - extension, and this is used for the uploaded image & thumbnail - names. - bucket - A core.s3.S3Bucket instance to upload to. - metadata - If not None, must be a dictionary of metadata to apply to the - uploaded file and thumbnail. - new_size - If not None, the image will be resized to the dimensions - specified by new_size, which must be a (width, height) tuple. - thumb_size - If not None, a thumbnail image will be created with the - dimensions specified by thumb_size, which must be a (width, - height) tuple. The thumbnail will use the same metadata, if - present, as the image. The thumbnail filename will be the - same basename as the image with a 't' appended. The - extension will be the same as the original image. - - A tuple is returned: (image_url, thumb_url) where thumb_url will be None if - a thumbnail was not requested. - """ - - logger.info('Processing image file: %s', filename) - - unique_key = make_key() - ext = os.path.splitext(filename)[1] - - # Re-orient if necessary - image = Image.open(filename) - changed, image = orient_image(image) - if changed: - image.save(filename) - - # Resize image if necessary - if new_size: - image = Image.open(filename) - if image.size > new_size: - logger.debug('Resizing from {} to {}'.format(image.size, new_size)) - image.thumbnail(new_size, Image.ANTIALIAS) - image.save(filename) - - # Create thumbnail if necessary - thumb = None - if thumb_size: - logger.debug('Creating thumbnail {}'.format(thumb_size)) - image = Image.open(filename) - image.thumbnail(thumb_size, Image.ANTIALIAS) - thumb = BytesIO() - image.save(thumb, format=image.format) - - # Upload images to S3 - file_key = unique_key + ext - logging.debug('Uploading image') - image_url = bucket.upload_from_filename(file_key, filename, metadata) - - thumb_url = None - if thumb: - logging.debug('Uploading thumbnail') - thumb_key = '{}t{}'.format(unique_key, ext) - thumb_url = bucket.upload_from_string(thumb_key, - thumb.getvalue(), - metadata) - - logger.info('Completed processing image file: %s', filename) - - return (image_url, thumb_url) diff -r bd594bcba5eb -r 4f265f61874b core/images/upload.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/core/images/upload.py Tue Sep 22 20:23:50 2015 -0500 @@ -0,0 +1,94 @@ +"""This module contains a function to upload an image file to a S3Bucket. + +The image can be resized and a thumbnail can be generated and uploaded as well. + +""" +from base64 import b64encode +import logging +from io import BytesIO +import os.path +import uuid + +from PIL import Image + +from .utils import orient_image + + +logger = logging.getLogger(__name__) + + +def make_key(): + """Generate a random key suitable for a filename""" + return b64encode(uuid.uuid4().bytes, '-_').rstrip('=') + + +def upload(filename, bucket, metadata=None, new_size=None, thumb_size=None): + """Upload an image file to a given S3Bucket. + + The image can optionally be resized and a thumbnail can be generated and + uploaded as well. + + Parameters: + filename - The path to the file to process. The filename should have an + extension, and this is used for the uploaded image & thumbnail + names. + bucket - A core.s3.S3Bucket instance to upload to. + metadata - If not None, must be a dictionary of metadata to apply to the + uploaded file and thumbnail. + new_size - If not None, the image will be resized to the dimensions + specified by new_size, which must be a (width, height) tuple. + thumb_size - If not None, a thumbnail image will be created with the + dimensions specified by thumb_size, which must be a (width, + height) tuple. The thumbnail will use the same metadata, if + present, as the image. The thumbnail filename will be the + same basename as the image with a 't' appended. The + extension will be the same as the original image. + + A tuple is returned: (image_url, thumb_url) where thumb_url will be None if + a thumbnail was not requested. + """ + + logger.info('Processing image file: %s', filename) + + unique_key = make_key() + ext = os.path.splitext(filename)[1] + + # Re-orient if necessary + image = Image.open(filename) + changed, image = orient_image(image) + if changed: + image.save(filename) + + # Resize image if necessary + if new_size: + image = Image.open(filename) + if image.size > new_size: + logger.debug('Resizing from {} to {}'.format(image.size, new_size)) + image.thumbnail(new_size, Image.ANTIALIAS) + image.save(filename) + + # Create thumbnail if necessary + thumb = None + if thumb_size: + logger.debug('Creating thumbnail {}'.format(thumb_size)) + image = Image.open(filename) + image.thumbnail(thumb_size, Image.ANTIALIAS) + thumb = BytesIO() + image.save(thumb, format=image.format) + + # Upload images to S3 + file_key = unique_key + ext + logging.debug('Uploading image') + image_url = bucket.upload_from_filename(file_key, filename, metadata) + + thumb_url = None + if thumb: + logging.debug('Uploading thumbnail') + thumb_key = '{}t{}'.format(unique_key, ext) + thumb_url = bucket.upload_from_string(thumb_key, + thumb.getvalue(), + metadata) + + logger.info('Completed processing image file: %s', filename) + + return (image_url, thumb_url) diff -r bd594bcba5eb -r 4f265f61874b core/images/utils.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/core/images/utils.py Tue Sep 22 20:23:50 2015 -0500 @@ -0,0 +1,118 @@ +""" +This file contains common utility functions for manipulating images for +the rest of the applications in the project. +""" +from PIL import ImageFile +from PIL import Image + + +def parse_image(file): + """ + Returns a PIL Image from the supplied Django file object. + Throws IOError if the file does not parse as an image file or some other + I/O error occurred. + + """ + parser = ImageFile.Parser() + for chunk in file.chunks(): + parser.feed(chunk) + image = parser.close() + return image + + +def downscale_image_square(image, size): + """ + Scale an image to the square dimensions given by size (in pixels). + The new image is returned. + If the image is already smaller than (size, size) then no scaling + is performed and the image is returned unchanged. + + """ + # don't upscale + if (size, size) >= image.size: + return image + + (w, h) = image.size + if w > h: + diff = (w - h) / 2 + image = image.crop((diff, 0, w - diff, h)) + elif h > w: + diff = (h - w) / 2 + image = image.crop((0, diff, w, h - diff)) + image = image.resize((size, size), Image.ANTIALIAS) + return image + + +# Various image transformation functions: +def flip_horizontal(im): + return im.transpose(Image.FLIP_LEFT_RIGHT) + +def flip_vertical(im): + return im.transpose(Image.FLIP_TOP_BOTTOM) + +def rotate_180(im): + return im.transpose(Image.ROTATE_180) + +def rotate_90(im): + return im.transpose(Image.ROTATE_90) + +def rotate_270(im): + return im.transpose(Image.ROTATE_270) + +def transpose(im): + return rotate_90(flip_horizontal(im)) + +def transverse(im): + return rotate_90(flip_vertical(im)) + +# From http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/EXIF.html +# EXIF Orientation tag values: +# 1 = Horizontal (normal) +# 2 = Mirror horizontal +# 3 = Rotate 180 +# 4 = Mirror vertical +# 5 = Mirror horizontal and rotate 270 CW +# 6 = Rotate 90 CW +# 7 = Mirror horizontal and rotate 90 CW +# 8 = Rotate 270 CW + +ORIENT_FUNCS = { + 2: flip_horizontal, + 3: rotate_180, + 4: flip_vertical, + 5: transpose, + 6: rotate_270, + 7: transverse, + 8: rotate_90, +} + +ORIENT_TAG = 0x112 + + +def orient_image(im): + """Transforms the given image according to embedded EXIF data. + + The image instance, im, should be a PIL Image. + If there is EXIF information for the image, and the orientation tag + indicates that the image should be transformed, perform the transformation. + + Returns a tuple of the form (flag, image) where flag is True if the image + was oriented and False otherwise. image is either a new transformed image or + the original image instance. + + """ + if hasattr(im, '_getexif'): + try: + exif = im._getexif() + except IndexError: + # Work around issue seen in Pillow + # https://github.com/python-pillow/Pillow/issues/518 + exif = None + + if exif and ORIENT_TAG in exif: + orientation = exif[ORIENT_TAG] + func = ORIENT_FUNCS.get(orientation) + if func: + return (True, func(im)) + + return (False, im) diff -r bd594bcba5eb -r 4f265f61874b core/tests/test_html.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/core/tests/test_html.py Tue Sep 22 20:23:50 2015 -0500 @@ -0,0 +1,68 @@ +"""Tests for the core.html module.""" +import unittest + +from core.html import ImageCheckError +from core.html import image_check + + +TEST_HTML = """ +

Posters and Facebook events are starting to come in...

+

image

+

image

+""" + + +class ImageCheckTestCase(unittest.TestCase): + def setUp(self): + self.allowed_hosts = ['example.com'] + + def test_happy_path(self): + url1 = 'https://example.com/1.jpg' + url2 = 'https://example.com/2.jpg' + html = TEST_HTML.format(src1=url1, src2=url2) + + result = image_check(html, self.allowed_hosts) + self.assertTrue(result) + + def test_empty_image(self): + url1 = 'https://example.com/1.jpg' + url2 = '' + html = TEST_HTML.format(src1=url1, src2=url2) + + self.assertRaises(ImageCheckError, image_check, html, self.allowed_hosts) + + def test_relative_ok(self): + url1 = 'https://example.com/1.jpg' + url2 = '/some/path/2.jpg' + html = TEST_HTML.format(src1=url1, src2=url2) + + result = image_check(html, self.allowed_hosts) + self.assertTrue(result) + + def test_non_https(self): + url1 = 'http://example.com/1.jpg' + url2 = 'https://example.com/2.jpg' + html = TEST_HTML.format(src1=url1, src2=url2) + + self.assertRaises(ImageCheckError, image_check, html, self.allowed_hosts) + + def test_missing_hostname(self): + url1 = 'http:///1.jpg' + url2 = 'https://example.com/2.jpg' + html = TEST_HTML.format(src1=url1, src2=url2) + + self.assertRaises(ImageCheckError, image_check, html, self.allowed_hosts) + + def test_hostname_not_allowed1(self): + url1 = 'https://xxx.example.com/1.jpg' + url2 = 'https://example.com/2.jpg' + html = TEST_HTML.format(src1=url1, src2=url2) + + self.assertRaises(ImageCheckError, image_check, html, self.allowed_hosts) + + def test_hostname_not_allowed2(self): + url1 = 'https://xxx.example.com/1.jpg' + url2 = 'https://yyy.example.com/2.jpg' + html = TEST_HTML.format(src1=url1, src2=url2) + + self.assertRaises(ImageCheckError, image_check, html, self.allowed_hosts) diff -r bd594bcba5eb -r 4f265f61874b requirements.txt --- a/requirements.txt Sun Sep 13 14:51:33 2015 -0500 +++ b/requirements.txt Tue Sep 22 20:23:50 2015 -0500 @@ -33,6 +33,7 @@ rsa==3.1.4 simplejson==3.6.5 uritemplate==0.6 +requests==2.7.0 # # These packages I punted on and hacked into my virtualenv by # symlinking to the global site-packages: diff -r bd594bcba5eb -r 4f265f61874b requirements_dev.txt --- a/requirements_dev.txt Sun Sep 13 14:51:33 2015 -0500 +++ b/requirements_dev.txt Tue Sep 22 20:23:50 2015 -0500 @@ -38,6 +38,7 @@ mock==1.0.1 lxml==3.4.2 testfixtures==4.1.2 +requests==2.7.0 -e git+https://github.com/gremmie/django-elsewhere.git@1203bd331aba4c5d4e702cc4e64d807310f2b591#egg=django_elsewhere-master -e git+https://github.com/notanumber/xapian-haystack.git@a3a3a4e7cfba3e2e1be3a42abf59edd29ea03c05#egg=xapian_haystack-master -e hg+https://bgneal@bitbucket.org/bgneal/queues@862e8846f7e5f5a0df7f08bfe4b4e5283acb4614#egg=queues-dev diff -r bd594bcba5eb -r 4f265f61874b requirements_mac.txt --- a/requirements_mac.txt Sun Sep 13 14:51:33 2015 -0500 +++ b/requirements_mac.txt Tue Sep 22 20:23:50 2015 -0500 @@ -43,6 +43,7 @@ paramiko==1.15.2 pss==1.40 wsgiref==0.1.2 +requests==2.7.0 # # These packages I punted on and hacked into my virtualenv by # symlinking to the global site-packages: diff -r bd594bcba5eb -r 4f265f61874b sg101/settings/base.py --- a/sg101/settings/base.py Sun Sep 13 14:51:33 2015 -0500 +++ b/sg101/settings/base.py Tue Sep 22 20:23:50 2015 -0500 @@ -295,6 +295,8 @@ USER_PHOTOS_MAX_SIZE = (660, 720) USER_PHOTOS_THUMB_SIZE = (120, 120) USER_PHOTOS_RATE_LIMIT = (50, 86400) # number / seconds +HOT_LINK_PHOTOS_BASE_URL = 'https://s3.amazonaws.com/' +HOT_LINK_PHOTOS_BUCKET = 'sg101.forum.photos' USER_IMAGES_SOURCES = [ 'dl.dropboxusercontent.com', diff -r bd594bcba5eb -r 4f265f61874b sg101/templates/forums/show_form.html --- a/sg101/templates/forums/show_form.html Sun Sep 13 14:51:33 2015 -0500 +++ b/sg101/templates/forums/show_form.html Tue Sep 22 20:23:50 2015 -0500 @@ -39,6 +39,20 @@ + +
{% csrf_token %}
@@ -48,10 +62,6 @@ form. After the photo is uploaded it will be resized and an image code will be placed in the post box, above.

-

To hot-link a photo hosted elsewhere, use the - - icon on the post box. -


limit +def rate_limit_user(user): + """Shared function to rate limit user uploads.""" + key = 'user_photos:counts:' + user.username + limit = settings.USER_PHOTOS_RATE_LIMIT + if rate_limit(key, *limit): + raise forms.ValidationError("You've exceeded your upload quota. " + "Please try again later.") + + class UploadForm(forms.Form): image_file = forms.ImageField() @@ -38,14 +49,7 @@ def clean(self): cleaned_data = super(UploadForm, self).clean() - - # rate limit uploads - key = 'user_photos:counts:' + self.user.username - limit = settings.USER_PHOTOS_RATE_LIMIT - if rate_limit(key, *limit): - raise forms.ValidationError("You've exceeded your upload quota. " - "Please try again later.") - + rate_limit_user(self.user) return cleaned_data def save(self): @@ -113,3 +117,55 @@ for chunk in fp.chunks(): md5.update(chunk) return md5.hexdigest() + + +class HotLinkImageForm(forms.Form): + """Form for hot-linking images. If the supplied URL's host is on + a white-list, return it as-is. Otherwise attempt to download the image at + the given URL and upload it into our S3 storage. The URL to the new location + is returned. + """ + url = forms.URLField() + + def __init__(self, *args, **kwargs): + self.user = kwargs.pop('user') + super(HotLinkImageForm, self).__init__(*args, **kwargs) + + def clean_url(self): + the_url = self.cleaned_data['url'] + self.url_parts = urlparse.urlsplit(the_url) + if self.url_parts.scheme not in ['http', 'https']: + raise forms.ValidationError("Invalid URL scheme") + return the_url + + def clean(self): + cleaned_data = super(HotLinkImageForm, self).clean() + rate_limit_user(self.user) + return cleaned_data + + def save(self): + import pdb; pdb.set_trace() + + url = self.url_parts.geturl() + if (self.url_parts.scheme == 'https' and + self.url_parts.hostname in settings.USER_IMAGES_SOURCES): + return url + + # Try to download the file + path = download_file(url) + + # Upload it to our S3 bucket + bucket = S3Bucket(access_key=settings.USER_PHOTOS_ACCESS_KEY, + secret_key=settings.USER_PHOTOS_SECRET_KEY, + base_url=settings.HOT_LINK_PHOTOS_BASE_URL, + bucket_name=settings.HOT_LINK_PHOTOS_BUCKET) + + now = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S') + metadata = {'user': self.user.username, 'date': now} + + url, _ = upload(filename=path, + bucket=bucket, + metadata=metadata, + new_size=settings.USER_PHOTOS_MAX_SIZE) + + return url diff -r bd594bcba5eb -r 4f265f61874b user_photos/urls.py --- a/user_photos/urls.py Sun Sep 13 14:51:33 2015 -0500 +++ b/user_photos/urls.py Tue Sep 22 20:23:50 2015 -0500 @@ -18,4 +18,7 @@ GalleryView.as_view(), name='user_photos-gallery'), url(r'^delete/$', 'user_photos.views.delete', name='user_photos-delete'), + url(r'^hotlink/$', + 'user_photos.views.hotlink_image', + name='user_photos-hotlink'), ) diff -r bd594bcba5eb -r 4f265f61874b user_photos/views.py --- a/user_photos/views.py Sun Sep 13 14:51:33 2015 -0500 +++ b/user_photos/views.py Tue Sep 22 20:23:50 2015 -0500 @@ -5,14 +5,14 @@ from django.contrib.auth import get_user_model from django.contrib.auth.decorators import login_required from django.http import (HttpResponse, HttpResponseForbidden, - HttpResponseNotAllowed) + HttpResponseNotAllowed, JsonResponse) from django.shortcuts import render, redirect, get_object_or_404 from django.views.generic import ListView from django.views.decorators.http import require_POST from django.utils.decorators import method_decorator from django.contrib import messages -from user_photos.forms import UploadForm +from user_photos.forms import HotLinkImageForm, UploadForm from user_photos.models import Photo from user_photos.s3 import delete_photos @@ -140,3 +140,48 @@ msg) return redirect(ret_view, username) + + +def hotlink_image(request): + """This view is responsible for accepting an image URL from a user and + converting it to a URL pointing into our S3 bucket if necessary. + + We return a JSON object response to the client with the following name + & value pairs: + 'error_msg': string error message if an error occurred + 'url': the image URL as a string if success + """ + ret = {'error_msg': '', 'url': ''} + status_code = 400 + + if not request.user.is_authenticated(): + ret['error_msg'] = 'Please login to use this service' + return JsonResponse(ret, status=status_code) + if not request.is_ajax() or request.method != 'POST': + ret['error_msg'] = 'This method is not allowed' + return JsonResponse(ret, status=status_code) + + if settings.USER_PHOTOS_ENABLED: + form = HotLinkImageForm(request.POST, request.FILES, user=request.user) + if form.is_valid(): + try: + ret['url'] = form.save() + status_code = 200 + except Exception as ex: + ret['error_msg'] = str(ex) + status_code = 500 + else: + # gather form error messages + errors = [] + non_field_errors = form.non_field_errors().as_text() + if non_field_errors: + errors.append(non_field_errors) + for field_errors in form.errors.values(): + errors.append(field_errors.as_text()) + ret['error_msg'] = '\n'.join(errors) + else: + ret['error_msg'] = 'Image linking is temporarily disabled' + status_code = 403 + + return JsonResponse(ret, status=status_code) +