Notes

Ingredients: 2/3 cup oats. 1 cup milk. 1 tbsp cacao powder. 1 tsp chia. A little bit of honey for sweetness and some dry fruits. Steps: Mix oats and milk, put on high heat and bring it to a boil. Bring down the heat to low, add rest of the ingredients, mix well and let cook for another few minutes until the texture looks good.

Anycast is a neat trick: the IP address you vend out to a client remains constant but their requests go to a datacenter closest to them. That way, you need to do rate-limiting for them within a datacenter and not across. The latter would be hard because you’d have to replicate data in real time. There are various algorithms for local rate limiting. Good ones seem to be token bucket, leaky bucket and sliding window. Factors to evaluate these algorithms: do they allow bursts and the size of parameters you need to track per client-id. Distributed rate limiting: Gossip between hosts vs hosts using a single distributed cache: the difference probably is just that we don’t have to deal with the former ourselves when we use the latter. In other words, we offload replication to latter instead of managing ourselves in former. One issue with distributed cache is that we need to ensure it scales with the service fleet size, so that we don’t have a large fleet overwhelming a small fleet. I find gossip difficult to visualize. Also, it has lot of chattiness. An alternative to gossip is to elect a coordinator/leader that reads data from all the hosts and updates them back with updated numbers. Such leader election may not have to be super-accurate. Consistent hashing may be a good idea to map key to host which stores the counters. So, service hosts know which host, out of a cluster of rate trackers, they have to go for a given client id. Probably think of consistent hashing whenever you want to deal with data placement.

D = durability. Easiest to understand. C = consistency, i.e. application-level constraints or invariants are met. Property of the application that’s using the DB and not the DB itself. So, kind of useless. A = atomicity or, better explained as, abortability. Either all commands in a transaction go through or none of them. I = isolation is the trickiest one. Concurrent operations in a DB can lead to various race conditions. Dirty reads: read something that’s not yet committed. Dirty writes: transaction T1 writes on top of partial writes, which aren’t yet committed, by another transaction T2. Read skew: Let’s say a long-running client reads a lot of records from a DB. If transactions keep changing records during its run, the state of the DB the client captures may not be consistent. Write skew: Let’s say a transaction checks for a premise and then writes based on that. What happens if the premise is no longer valid by the time the write happens? DB isolation features: Read committed: doesn’t allow dirty reads or writes. Snapshot isolation or repeatable reads: no read skews. Need to track transaction level information per record to implement this. Serializable: no write skews. Strongest level but isn’t cheap to implement. So, not necessarily the default amongst all the DBs. Algorithms to implement: Literally serialize on the DB. Can work if all transactions are guaranteed to be fast. 2PL: lock all rows that match the premise before updating them. So, pessimistic locking. SSI, i.e. snapshot serializable isolation: Allow all transactions to run concurrently but don’t let them commit if they’d introduce a conflict. So, optimistic. 2-phase commit and 2-phase locking, i.e. 2PC and 2PL are different things. You use 2PC to support transactions across multiple systems. (I still need to learn about those in more detail.) Open questions: ...

Man, my phone and laptop addiction is back and it’s possibly bigger than ever. At 1:45 am, I thought I am finally done with my phone, only to pick it up exactly 3 minutes later to write this up. I’ll fix this now. I’ll stop idle internet browsing on both the devices. If I still want to, I’ll first create a list on what all I want to browse and then stick to that. Rest of the time, I’ll rather listen or read books, study or plan for the future. ...

Saw this email at work: I thought I’d share the process that worked for my parents. My parents received Covishield in India earlier this year and now they are in Seattle. As a first step, I reached out to my general physician to get her advice on if they can/should get the booster in the US and my doctor said yes, they should. We were able to get them a Pfizer booster shot at the Costco Pharmacy in Woodinville (walk-in, on a weekday). We took the following documents to prove they were eligible for a booster in the US: ...

Two types of companies: those that outsource operations to someone else vs those that manage themselves. Latter should be better but they won’t support a lot of locations. If we move abroad, better to choose a location that doesn’t have state income taxes, otherwise use same state as ours. Good services: https://travelingmailbox.com/ - seems to be the most bang for the buck. You can probably mark mails as junk after they are scanned and they won’t count towards monthly limits. Also, 2 months per year free. https://www.virtualpostmail.com/ https://www.earthclassmail.com/ https://sbimailservice.com https://www.usglobalmail.com

A few months back, I realized that I will learn deep learning better by working on real world projects as opposed to just doing exercises from a book. Then, when I wanted to go deeper into distributed systems, I decided to implement the algorithms to make them stick. That wasn’t a bad idea. For example, I implemented consistent hashing and hierarchical timing wheels and it was a great way to learn. However, coding takes effort and I kept putting off more stuff, such as rate limiting or consensus algorithms, because of the heavy cognitive effort. In such situations, I get rigid about how I want to do something. When that idea stops working, I continue to stick to it at the cost of making further progress. ...

The problems of distributed state machines and distributed logs are essentially the same. 3 roles: leader, follower and candidate. Only 2 RPC APIs in the algorithm. RequestVotes and AppendEntries (which also serves as heartbeats from leader to followers). Terms are tied to leadership changes, not with time. There can be terms that don’t have any leader because of election failures. Log indices change with commands from clients. So, multiple indices in sequence could have the same term. Leader is always right. So, it can change old log entries on other nodes. However, during election, the algorithm tries to elect a leader that is most up to date. Invariant: if, at an index, term number is the same between two nodes, everything before it should also be so. If nodes find discrepancies during replication, they reject the AppendEntries call from the leader and it’s the latter’s responsibility to retry for smaller indices. So, just those two APIs in the system. (Followers don’t make any API calls to resolve discrepancies.) During election, node will vote for a candidate only if the latter has a more complete log, which is defined by: 1) higher term and 2) higher log index, in sequence. 2 different majorities will contain at least 1 common member. While obvious, this has an important implication. If 2 different leaders are elected in sequence, at least 1 node will be common between those that voted for them. Open questions: ...

Paxos Paxos is notorious for being difficult to understand. However, the vanilla version seems somewhat straightforward albeit with one limitation: it only allows you to reach consensus on one value. For practical purposes, you want to reach consensus for multiple values in sequence, similar to a replicated log, and you need multi-Paxos for that. And that is hard. Think of basic Paxos as: “do we know already where we’ll go for dinner” and, if not, “let’s go for burgers”. Three roles: proposers, acceptors and learners. Odd number of acceptors. While both proposers and acceptors use numbers in the algorithm, those are just mechanisms to reach consensus on one value. Once consensus is reached, they can continue using more numbers but that value will never change. So, one run of the algorithm gets you one consensus value. If you want more, use multi-Paxos. The algorithm, the way I understand it, is as follows: ...

The following is a copy of the basic tutorial that rsync.net sent over to me when I signed up for their service. The login information above provides everything you need to access your filesystem, however some examples may be helpful: scp /some/file [email protected]: rsync -avz /some/file [email protected]: Yes, the trailing ':' at the beginning of the remote path IS necessary. Note in these two examples that you do NOT need to specify a home, or "starting" directory. If you wish to specify the location of a remote directory in your rsync.net account, you do it WITHOUT a preceding '/': scp /some/file [email protected]:some/directory/tree OR: rsync -avz /some/file [email protected]:other/dir/tree If you are simply connecting via FTP or SFTP, nothing is needed but the username and hostname: ftp [email protected] sftp [email protected] ----- This email has been sent in clear text, so you should immediately change your password. UNIX, Linux and Mac OS X users can do this in the terminal: ssh -t [email protected] passwd Windows users may set their password the same way, using the 'plink' command on the command line: https://www.rsync.net/resources/howto/windows_plink.html Please also note that the password for the web based login (mentioned below) needs to be updated separately from within the Account Manager interface. Your Account Manager and storage account passwords are not linked. ----- Configuration and settings of your account, including: account size, geo-redundancy, idle alerts and billing can all be managed with our web based account manager: https://www.rsync.net/am/dashboard.html?u=de1682 You can also view your quota and run simple UNIX commands over SSH: ssh [email protected] quota For a list of all commands that you can run over SSH, please see: https://www.rsync.net/resources/howto/remote_commands.html Windows users can run these same commands using the 'plink' command: https://www.rsync.net/resources/howto/windows_plink.html ----- Your home directory contains a .ssh directory that you can place an authorized_keys file in, which will allow you to interact with your filesystem in an automated fashion. There is a detailed HOWTO on this topic here: https://rsync.net/resources/howto/ssh_keys.html ----- rsync.net automatically creates and maintains 7 days of snapshots, or versions, for you. If you require more than 7 days (or 7 days + 4 weeks for 10+ TB accounts) please contact [email protected]. You can read more about our filesystem snapshots here: https://www.rsync.net/resources/howto/snapshots.html ----- If you are backing up data to your filesystem directly from a NAS, please contact us first so that we may properly configure your filesystem to work with your NAS. ----- If you need to use borg 1.x we recommend running your command as follows: borg COMMAND --remote-path=borg1 ... No special options are required to run borg 0.x ----- You can verify our SSH key fingerprints here: https://www.rsync.net/resources/fingerprints.txt ----- Basic usage examples can be found at https://www.rsync.net/resources/index.html Again, we are very happy to have you as a customer and look forward to serving you in any way that we can. -- The rsync.net Team - [email protected]