Try   HackMD

Finding Sites hosted on GitHub Pages

What?

There have been several reports of sites that are hosted on GitHub Pages being blocked by Airtel. These are mostly sites that have been fronted with CloudFlare Flexible SSL, since that means Airtel (as CloudFlare's network vendor) can MITM the CloudFlare<>GitHub Pages connectwion and block the site.

To validate this hypothesis, I want a dataset of all sites hosted on GitHub Pages. With this in hand, it becomes much easier to find what's hosted on CloudFlare, what's hosted directly on GitHub, where's the SSL termination happening, and whether it is blocked.

But to get there, I need a dataset of ~almost all sites powered by GitHub Pages using a Custom Domain.

I came up with several approaches, but haven't found anything that works.

Search using GitHub API

The GitHub API returns 302k results for CNAME files, but trying to access this over the API hits secondary API limits very quickly (like on the second request).

Censys.io

I have a research account on Censys, but Censys scans each host, and does a reverse DNS lookup after. And it limits number of "virtual-hosts per IP" to 500 or so. With the 10 or so IPs fronting GitHub Pages, it's not representative.

Certificate Transparency Logs

Both censys.io as well as crt.sh have this data, but there's nothing indicative in the certs which would point to it being hosted by GitHub Pages. GitHub issued TLS certs are from LetsEncrypt, while CloudFlare uses its own CA.

Scrape a search engine

No indication in the result to search for, again. Google doesn't let me search by response headers.

Lookup by headers

For every site hosted on GitHub, the server returns a unique set of headers that is fairly indicative:

X-GitHub-Request-Id, X-Fastly-Request-ID, Server: GitHub.com. For sites fronted with CloudFlare, we get server: cloudflare and a few other CloudFlare headers.

So if I wanted to search for sites on GitHub Pages fronted with CloudFlare, I would want to look for sites that return a X-GitHub-Request-Id header along with server: cloudflare.

The common-crawl dataset sounds like the best approach to get this, but I'm unsure about coverage, and was wondering if there are better ways to solve this? Is there a search engine where I could search by response headers?

Last changed by 
 
nemo
0
205

Read more

Reverse Engineering Project - Uniken Security

Introduction Uniken Security is an Indian company used by many banks and financial firms to protect their end-user applications by a product called REL-ID. There is unfortunately very little research literature available about the product or data about how it works. REL-ID REL-ID is a end-user security product that does end-device data collection to detect malicious activity to slow down attackers. Among it&apos;s various claims:

Dec 13, 2021
100 din mein paisa double

Introduction Last time we met, I said &quot;Skills &gt; Learning&quot;. And skills of all kinds count, no matter what they are. I said &quot;Financial Independence&quot; should be your north star, but I didn&apos;t do a good enough job explaining what I meant. So, this is me putting my money behind this and getting you to learn some money-making skills in the next 3 months. What You try to earn some money in the next 100 days. You get a bonus payout (from me) of $x^2/25000$ INR where x=money you earn within 100 days of accepting this challenge. Here&apos;s a nice table (first column is money you earn on your own, and second column is money I put in): The closer you get to 25,000, the closer you get to &quot;doubling your money&quot;.

Nov 9, 2021
AWS gameday - Harshil/Nemo

Ideas: Figure out a way to automate the changes to DynamoDB table

Aug 7, 2020
10: Katamari Damaci

On one of my recent livestreams, this topic came up. (By the way, we do livestreams now&#x2014;though ironically for this topic, they do not involve me playing video games. Instead, they&#x2019;re full of exciting things like me signing books. Find the past ones on my YouTube channel.) My goal with this section of the newsletter is to let you get to know me better. I thought it would be a fun experiment for me to talk about my favorite video games. You shouldn&#x2019;t consider this a &#x201C;best of all time&#x201D; list, though I do recommend these games. I&#x2019;m not saying these are the best; merely they are my favorites. Think of this list more as an insight into who I am and the things I like. Since picking favorites is generally tough, I decided to weight the list toward a few factors. First, if the game is part of a series, I decided I should generally like the whole series to include it here. I&#x2019;ll mention my personal favorite game for each item, but in general, this is a &#x201C;favorite franchise&#x201D; list as much as it is a favorite game list. Second, the game or franchise should be one I&#x2019;ve gone back to multiple times throughout my life, rather than being just something I loved when I was young. Nostalgia will play a part in the list, but I want it to be more than just &#x201C;teenage Brandon&#x2019;s greatest hits.&#x201D; Finally, I&#x2019;m eliminating games I or my friends made&#x2014;so no Infinity Blade, for example. I should also note to some of my younger readers that some of these may not be appropriate for all ages. There are a couple that even I played with a few mods to tone down some of the graphic content. (I really don&#x2019;t need to see that many head explosions, Fallout. Really, I&#x2019;m good.)

Jul 7, 2020
Read more from nemo
Published on HackMD