Selfelected

Cosmosdb change feed and deleted items

Until Microsoft implements change feed notifications for deletions we have to soft delete items in Cosmosdb if we use change feed notifications.

TL;DR

Cosmosdb has a handy feature where it journals changes which can then be read in another process. My use was for updating an Azure search index (Elastic search variant) whenever I made a change.

But… how do we handle deletions?

Hidden in the documentation is the information that ,,albeit all changes (within a shard) is guaranteed to be in chronological order it is always the last (present) item handed to the change feed process. So when an item is deleted there is no item and, due to implementation, also no event.

Remedy 1: Soft delete the item by marking it as “deleted”. This is a common approach in many a system.
A drawback is that the repository will end up with stale data.

Remedy 2: Soft delete the item as above. Then make sure this information has been conveyed (in my case the item should be deleted from the search database) before properly deleting it.
This is a more complex solution where the logic still has to handle soft deleted items while also adding the complexity of yet a process to scan(?) the repo for items to delete.

Your turn

I have found no uservoice, github or azurefeedback item for voting on getting a delete event; the closest is an item for getting a event for it in Azurefunctions.
In lack of better place I voted there; so can you if you have the need.

Update

I wrote a suggestion of my own here at azurefeedback.
I did try to find an existing suggestion but failed. Am I the only one deleting data? Or the guys at Microsoft will find an already existing suggestion where I can move my votes.

Update

The functionality is on its way
“Support for deletes on Change feed is planned. Do not have an exact ETA but we are targeting some time in 2020.“

Possibly possible to install Powershell with one line of code

Scott Hanselman wrote about installing Powershell as a global tool here.

It goes something like:

dotnet tool install --global PowerShell

Update

As Windows10 nowadays comes with Powershell this is not so much of a problem any more.

Dotnet framework design guidelines

There is an MSDN article a book on the subject but since it is more than 10 pages only few have read it.

Alas the Corefx team created a short list with stuff here: https://github.com/dotnet/corefx/blob/master/Documentation/coding-guidelines/framework-design-guidelines-digest.md

It is much easier to read

Comments about rules I have found but often not followed

“Do not postfix enums with Enum.” To be honest – I have often found it the best solution to postfix with Enum.

“Do not organise namespaces after company hierarchy.” Even though Conway’s law is a thing, a system should not keep the history of a changed organisation chart.

“Do not use List<T> on public methods.” List<T> is a complex beast. But using IEnumerable<T> might have some reiteration performance impact, that is when you have to loop through the same collection twice.

Other comments

“Carefully choose names for your types, methods, and parameters.” Naming things can be hard so take your time and discuss with your colleagues.

“Do not create mutable value types.” I totally agree. It is easier to debug imutable. Often when I track down a but I have only a (bad) call stack and the name of a public method and some source code. Then it is very important the code is easy to follow by hand.

“Do not throw

“Members returning arrays should return copies of an internal master array.” I is good OOP practice to not publish internal stuff. Correction, it is cruical OOP practice to not expose the innards of a class.

Other other comments

“Prefer classes over interfaces.” Why? Context please.

IMO the linked article forgot to mention that public methods should not have optional parameters as they are compiled into the caller and not callee.

Avoid public as any change to them might break the API. Unfortunately Visual studio defaults new classes (and methods?) to public so I see them everywhere.

Experimenting with setting up a new machine every third month

I am currently experimenting with using a virtual machine for my development. I have earlier done it through OSX/Parallels but now I am trying a Win/Win solution with Hyper-V.

Every third month, that is for how long the license is valid, I do a Hyper-V Manager->Quick create->Windows 10 dev environment.

It comes with Visual studio, Visual studio code, Powershell 6 and some dotnet preinstalled. All I have to do is start VS and tell it to update itself..

Then I need Git, Notepad++, VSCode, Keyboard1337 and BecerroInamovible. Ergo I need Chocolatey too.

Here is how I do it

Fire up a console with elevated priveges.

Window-button, cmd, ctrl-shift-enter. Enter.

1
Window-button, cmd, ctrl-shift-enter. Enter.

Install Chocolatey.

1
@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"

Restart console.

I have checked that all chocolatey packages I reference are “trusted package”. I cannot know if they packages remain “trusted package” at the time you are reading this. The choice is yours.

1
2
Exit. Enter.

Window-button, cmd, ctrl-shift-enter. Enter.

Install Git.

1
choco install git.install

Install Notepad++.

1
choco install notepadplusplus.install

Install Posh-Git

1
2
PowerShellGet\Install-Module posh-git -Scope CurrentUser -AllowPrerelease -Force

Add-PoshGitToProfile -AllHosts

Install Powershell core

1
choco install powershell-core

Install Keyboard1337.

I believe Keyboard1337 is a zip if you git clone BecarroInamovible. If so, you don’t have to download Keyboard1337.

Download Keyboad 1337.zip. Unblock. Unpack. Install.
Run Keyboard 1337\l337\setup.exe (yupp. there’s a typo there)
Language preferences -> Keep English(Sweden) and English(United States). The latter is Keyboard1337

BecerroInamovible

It has a dependency on Autohotkey.

1
choco install autohotkey

Copy contents of https://raw.githubusercontent.com/LosManos/keyboard1337/master/BecerroInamovible.ahk to C:\DATA\PROJEKT\Keyboard1337

As an alternative do “git clone https://github.com/LosManos/keyboard1337.git”

As short as possible:

choco install git.install -y & choco install notepadplusplus.install -y & choco install autohotkey -y & choco install powershell-core

1
choco install git.install -y & choco install notepadplusplus.install -y & choco install autohotkey -y & choco install powershell-core

Start a new console

1
2
PowerShellGet\Install-Module posh-git -Scope CurrentUser -AllowPrerelease -Force

Add-PoshGitToProfile -AllHosts

In a cmd (not powershell):

1
2
3
4
5
6
mkdir c:\DATA

mkdir c:\DATA\PROJEKT

pushd c:\DATA\PROJEKT

git clone https://github.com/LosManos/keyboard1337.git

"c:\Program Files\AutoHotkey\AutoHotkeyU64.exe" c:\DATA\PROJEKT\keyboard1337\BecerroInamovible.ahk

popd

The three reasons to use an O/RM

1. Type security
2. Data copying Entity/Recordset
3. Faster turn around with a moving schema

If these 3 issues aren’t a problem in you case, be reluctant to use an O/RM as an O/RM brings a bunch of other problems.

Type security

When you get the Customer.Verified out of a database the code already knows it is a boolean. No explicit type casting or declaring is needed as it is done elswhere, once and for all.

Now this isn’t totally true but depends on whether LINQ or HQL (or for another O/RM) you are using. But basically it is true, because of solid Left-right copying

Data copying

Copying data from a recordset to the Entity is done by the OR/M and configured once and for all. Unlike a programmer it won’t forget to copy a field in an obscure corner in your code base.

It isn’t hard to write a data copying function for this simple case and dragging a whole OR/M into the project is a lot of dependencies for little use.

Turn around

When the scheme is changing fast, like in the beginning of a project, or when prototyping, an O/RM is of great help to change in one place and let the coding tools and compiler do the rest of the changes in the project.

Schemes tend to solidify after a while and only change in small increments so speeding up the first 3 months for having a rucksack the coming 6 years might not be the best solution.

Powershell special characters and tokens

It is hard to web search for Powershell’s interpretation of $?, |, –% or whatnot but Neolisk has created a nice list.

Read here: http://www.neolisk.com/techblog/powershell-specialcharactersandtokens

Classes

Concise: https://xainey.github.io/2016/powershell-classes-and-concepts/

Watch Your code become intermediary C# before becoming CLR

Go to https://sharplab.io and start typing away in the left screen. The internediary code or IL is presented in the right.

The code you write can be in C#8, F# or VBnet. The output can be the input (try `using` to see how it is just a shorthand for `try…catch`).

The output is C#, F# and Vbnet; but you can also choose IL, JITAsm, the syntax tree and some more.

Use it to get a better understanding of what is produced by your code.

It is all open source https://github.com/ashmind/SharpLabhttps://github.com/ashmind/SharpLab.

Why I try to not use the debugger

Readable logs

In production we usually rely on logs to track down an error.
Even worse; in production we have more data and more logging to sift through than in development. The trees can be lost in the woods. If we are more unlucky we also have a time restraint.

To learn how to read a log file we must train. This means reading log files. And to read the we must first write to them; in a readable way.

So I write log files and read them to find my errors already in development. Training for production.

Learn to write

Too little information is bad. This is the usual case. Too much information is bad too as we cannot see the flow of the program due to uninteresting information. Writing the exact amount of information for every time is probably not doable but by training I learn and get better at it.

Readable code

If we develop the application through setting breakpoints and inspecting variables we are in a bad spot when production error happens as we seldom can do that.
All we have is a log with a stack and possibly a rough idea of the input.

If I instead learn to read the code and trace the execution path manually, already in the development phase, I am in a much better position when production hits the proverbial fan.

Startup time

Connecting the debugger takes time. Just starting the program is faster.

I only connect the debugger when I have to.

Powershell pitfalls

I would expect the above to return

but no. Instead

is returned.

Why?

It’s because a method call in an if statement has to be surrounded with paranthesises like so:

Now the output is

Powershell for devs, part 2

Continuing from Part1.

New to Powershell? Let me jot down a few good-to-know things. Some you already know. Some might save you hours of googling.

Below is a simple module. I will explain it row(s) by row(s).

Use Set-StrictMode. See part1 of this blog series.

This type of comment right before a method is recognised by Powershell and ends up in Get-Help.  Adhering to explaining the intention for your methods is considered good practice.

Inside the method is

This is what the parameters look like. One can set if a parameter is mandatory, some help text to be picked up by your favourite text editor, the [type] and default value.

A variable is set to a string. Other scripting languages send strings around. Powershell sends proper objects. The Get-Date-Format converts the DateTime value to a string.

Built into Powershell is the possibility to call (almost) anything with a -Verbose flag. Only then is the Write-Verbose called. Like a simple logging level.

Nothing strange here at first sight. Until you exeute in a console. Then you realise this string is outputed; because it is not inputed into something else, like a variable.

Also “$($variable.Method)” is the way to call methods inside a string.

Powershell is said to be able to use Linq. This is technically true but the syntax is so weird that I have never used it. This code though has (almost) the same behaviour and is easy to read.

GetPeople is a method call. Backtick concatenate lines and circumvents that Powershell has automatic statement ending with a line end. | pipes object and not strings. where is an alias for Where-Object. The rest is… Linqish.

Write-Host an object like $foundPerson output the contents of the object. Not just the type as in C#.
Write-Verbose “Stop:$(Get-Date-Format ‘t’)”
This string is outputed only if the method call is made with -Verbose. Plus an exmple on how to write a method call in a string.

Finally nothing surprising. Except that return can be left out to make the code harder to read.

@ tells Powershell to create a list of key-value pairs. Also often called a hash list.

Note the comma character. That makes $people an array of key-value pairs.

Only exported mehods are visible outside the module. It is like making them public.