Exempt CSRF check for a specified list of origins. am I doing right?

[wonderbeyond]

Firstly, I insist on using session-based authentication instead of some token based authentication.
Because in commonly-used token based authentication, the JavaScript code can always read the access token, which is XSS vulnerable.

The "SessionAuthentication" authentication scheme uses Django's session backend, and a django session is established by using a httpOnly cookie which can not be accessed by js code. So using such a session backend can avoid XSS vulnerability.

However, I want my REST API can be accessed from a specific list of origins, especially in testing environment.
So I made a custom authentication scheme CustomSessionAuthentication as below:

import re

from rest_framework.authentication import SessionAuthentication
from corsheaders.conf import conf as cors_headers_conf


def origin_found_in_csrf_exempt_list(origin):
    if origin in cors_headers_conf.CORS_ORIGIN_WHITELIST:
        return True

    for domain_pattern in cors_headers_conf.CORS_ORIGIN_REGEX_WHITELIST:
        if re.match(domain_pattern, origin):
            return True

    return False


class CustomSessionAuthentication(SessionAuthentication):
    """
    Works some like rest_framework.authentication.SessionAuthentication,
    but ignore csrf checking for **ajax** requests that come from a specified list of origins.
    """

    def authenticate(self, request):
        """
        Returns a `User` if the request session currently has a logged in user.
        Otherwise returns `None`.
        """

        # Get the session-based user from the underlying HttpRequest object
        user = getattr(request._request, 'user', None)

        # Unauthenticated, CSRF validation not required
        if not user or not user.is_active:
            return None

        http_origin = request.META.get("HTTP_ORIGIN")
        http_referer = request.META.get("HTTP_REFERER")

        if not http_origin and http_referer:
            http_origin = re.match(r'(^\w+://[^:/]+(:\d+)?).*$', http_referer).group(1)

        if not origin_found_in_csrf_exempt_list(http_origin):
            self.enforce_csrf(request)

        # CSRF passed with authenticated user
        return (user, None)

# in settings.py
...
ALLOWED_HOSTS = ['*']  # do not make this in production!
INTERNAL_IPS = ["127.0.0.1"]
CORS_ALLOW_CREDENTIALS = True
CORS_ORIGIN_REGEX_WHITELIST = [
    r"^https?://(127\.0\.0\.1|localhost)(:\d+)?$",
    r"^https?://172\.\d+\.\d+\.\d+(:\d+)?$",
]
...

Note I take corsheaders's origin whitelist configurations as my exemption list directly.

Am I doing right? Will I introduce some security holes?