Bitcoin Lesson | Mining

Bitcoin Lesson | Mining

In this presentation I’m going to cover how Bitcoin Mining works. I’ll start with “What is Mining?” And I’ll cover what it is, and why it exists. And then after that I’ll cover “How it Works”. But before I can cover how it works, I need to explain what a Hash Function is. Because that’s an integral part of how bitcoin works. Then I’ll cover the basics of how it works. And then I’ll go through it again, but I’ll explain it in a more technical way so if you’re a programmer, you’ll find that more interesting. Then after that I’ll start doing some examples and I’ll try and show you how you could actually mine a live block, in a very small amount of code. So that’s what the presentation is going to be. So I’ll get started with what Mining is. So, what is Mining? Imagine this is the Bitcoin Network. And these are all the computers, connected together, running the Bitcoin program. And sharing the same file, which is the blockchain. The way bitcoin works… If you want to send a bitcoin to someone, or transfer ownership of a bitcoin to someone else the way you do that is to create a transaction which is just a little line of data and then you’ll insert that in to a Node on the network. And then, if we didn’t have mining the way this would work would be that this computer would write it straight to the blockchain, the shared file. And then they would pass on that transaction, that piece of data on to the nodes they are connected to. and they will write it to their file and so on… until that transaction has propagated the entire network and written it to their shared file. So that would be a simple way of having a shared file, or a shared “ledger” of transactions. Then, when everyone has updated their copy of the file the ownership of that bitcoin has changed from one person to the next person. And that would be a simple transaction. But there is a problem that needs to be solved if you were to do it this way. Let’s color this man in green… What you can do is create a transaction to send the bitcoin to this person and insert it in to a node on this side of the network. But what you can do, because it’s a network of connected computers you can create a second transaction that sends the very same bitcoin to a different person. So you have just inserted two separate transactions in to the network. Trying to spend the same bitcoin. You can do this because it’s a network. So what will happen from there is these two transactions will start propagating the network and the computers on this side, as you can see they have received this red transaction the second one But the computers on this side they have received the green transaction. When this computer tries to pass on their green transaction on to this computer this computer has already received the red transaction spending this bitcoin so they would reject it. Which is good, but you still have the problem then of these two conflicting transactions on the network. Or basically, if you look at the network as a whole, who does this bitcoin belong to? If you were to run bitcoin this way and write transactions directly to the file you could create this double spend problem. So basically, Bitcoin solves this problem of not having conflicting transactions written to the blockchain, to this “shared file”. And that is what mining is. The way bitcoin solves this …and this is what makes bitcoin interesting… because we’ve had this technology before… *PROBLEM WITH NEVILLE’S MIC* So Bitcoin solves this problem of writing two conflicting transactions to a shared file. And this is what makes Bitcoin interesting and this is what I suppose the “invention” of Bitcoin is So the way Bitcoin solves this double spend problem is that it gives every computer running the Bitcoin program some temporary memory called the “Memory Pool” So what will happen is, you can’t stop this double spend attack but what will happen is, the person will insert the transaction in to this computer this green transaction And then they can go over here, and do the same double spend attack again and insert this second red transaction in to this computer. But obviously, these two transactions aren’t written directly to the file they get stored in temporary memory first So the same thing happens; both transactions propagate the network. This one will reject the green one because they have already received the red one And so there we are both transactions are on the network but they are not written to the file yet, so the file is still secure. What will happen then is all of these computers will work to try and get the transactions from their memory pool on to the file So they are all competing basically, to be the first one to get their memory pool transactions added on to the top of the file So let’s say this one is the first one that’s able to do it They add their transactions from their memory pool on to the file, like so… And when they have done that They will pass on their updated copy of the file to everyone else they are connected to And they will update their copies and this node here because they’ve got this conflicting green transaction in their memory pool but they have just received this red transaction spending the same bitcoins they will kick out this green transaction from their memory pool So this memory pool was like a waiting area, or a clearing house and they have received this red transaction in the updated file, so they will kick out this conflicting one. And then this node will pass on the updated copy to these nodes also and they will do the same thing, they will kick out any conflicting transactions from their memory pool So that’s how you solve the problem of having two conflicting transactions on the network. You just literally have a waiting area and all of the nodes compete to try and add their transactions on to the file. And there we are, this person now has the bitcoin. So basically this process of every node competing to try and get their transactions from their memory pool on to the file is called Mining. And that’s what I’m going to explain now… getting these transactions on to here. Any questions about that so far? NEVILLE: Not really, but one thing to note here is that every node on the network is not necessarily mining. That’s true, yep, yep. I just made a very simple diagram, trying to… yeah. Some of these might not be mining you can just run a node that will relay the blocks and transactions In this particular simplified diagram everyone is trying to mine. NEVILLE: So let’s say we have a node that is not mining, so if it gets a block… NEVILLE: which has a transaction and a conflicting transaction in the memory pool, it will kick it out. Yes, all this relaying of blocks and transactions, a node will do this but a miner just does this latter part of trying to add the transactions from their memory pool, that is a miner, this part here. But, you can be a node, but you don’t have to mine. Is that alright? So that’s the intro done. So the next part is explaining what this hash function is So that I can then cover how bitcoin mining works. Right, so a “Hash Function”. A hash function is just a small computer program that is quite useful. The way it works is, you can have some data so this is the letter “a” and the number “1” and you can insert it in to the hash function and the hash function will basically scramble it and produce a random string of data as a result. The useful property about a hash function is that you can put the same data in to it and it will always spit out the same result. So it will scramble it in the same way to produce this random string of data. And this string of data will always be the same size which is 32 bytes, or 64 characters because there are 2 characters per byte. If you were to make a tiny adjustment to the data you are putting in to the hash function so from here we’ve gone from “a1” to “a2” just a tiny adjustment Put that in, and the result will be completely different So we’ve only made a tiny adjustment, but the result of the hash function is wildly different And again, put “a3” in to it tiny adjustment again, but the result is completely different again. And an important part about a hash function as well is that if you don’t know what the result is beforehand there is no way you can predict what “a2” is going to spit out So if you don’t know what it already is, there’s no way you can figure out what the result is going to be which is useful as we’ll come to see in a moment Also, lastly, even though this is a random string of data it’s got the letters; “e”, “f”, “a” basically “a”, “b”, “c”, “d”, “e” and “f” in there these can actually be converted in to numbers and that’s because we are used to using the decimal format for numbers which uses the numbers 0 to 9 But you also have a number format called “hexadecimal” which is the same numbers but with also “a” “b” “c” “d” “e” and “f” And when you convert them over, it looks like this. So, this is called decimal because it uses 10 digits and “Deci” means 10, I think, in Latin. Then “Hexadecimal”: “Hexa”, like a hexagon, 6 sides. “Deci”, 10. Six, ten… Sixteen. So basically, this is what we use as humans but computers are very happy to read numbers in hexadecimal format So basically, with a hash function, you put data in and it spits out a random number as a result. So if you see a string like this in hexadecimal format just remember that it is a random number. Does that make sense, everything make sense there? Brilliant. So that was a Hash Function. Just a little tool that spits out random data If I quickly show you I don’t know if you can see this, if you can read the text A hash function is available in lots of programming languages So for example in PHP: There’s a function called hash() And then you can tell it which hash function you want to use So, SHA256 That is the one that’s used in Bitcoin Which stands for Simple Hashing Algorithm 256 Or 256 bits, which is the size of the result it spits out. And then you just put what you want to hash in to it That’s gone wrong… There we are. So that’s the result there So, you can use a hash function in any programming language I’ve just shown how to use it here in PHP In Ruby, just to be complete It’s included in the ‘digest’ library So you can go: `puts Digest::SHA56` then we want a hexadecimal digest… “Digest” is just another term for the result that gets spat out from a hash function So I’ll put “a1” in to there And there we have the same thing So a hash function is available in lots of programming languages Now I’ll move on to the basics of how Mining works So I’ll just run through giving a quick example of how this process here, of getting transactions from the memory pool on to the blockchain, works. Right, so, basics of Mining. Here we have the network again all the transactions in the memory pool *PROBLEM WITH NEVILLE’S MIC* So we are just going to look inside this computer This node here is a miner so we’re going to look inside it and see how it mines the transactions from its memory pool on to its file. *PROBLEM WITH NEVILLE’S MIC* So this is looking inside a Bitcoin Miner This is the memory pool It has got the red transaction in there But with a memory pool you can have much more than one transaction in there So let’s say this memory pool has got quite a few transactions already in there So what a miner will do They will create a container called a “Candidate Block” which is just a container for transactions So what they will do, they will fill this candidate block up with transactions from their memory pool like so and then they will try and add this block of transactions on to the file on to the blockchain So how do they get this Candidate Block they have built on to the blockchain? Well the first thing they do, they construct what’s called a “Block Header” Which is just a summary of all the data inside the block all the transactions in the block So in that block header you’ll have something called a “Version” which is just a version number for the block for the structure of the data So let’s say that’s 1. They will have some information called the “Previous Block” or the “Previous Block Hash” And the way that works is… Let’s say this is the blockchain This candidate block they are building It has to build on top of one of these blocks So what a miner will do is they will include the hash or the “Block Hash” which is like an ID number for a block of the one they want to build on top of Like so So in here, this is the block hash for this block So this just says which block they want to build on top of because every miner wants to build on to the top of the blockchain “Merkle Root” Now a Merkle Root is just the… Basically if you hash all the Transaction IDs together over and over again you will end up with a single hash at the end and this is called the Merkle Root. Like so. And basically this is like a fingerprint, like a digital fingerprint that summarizes all the transactions in the block. So again, this block header creates a unique fingerprint for all the data in the block but using a lot less data than all of the transaction data combined NEVILLE: Maybe if you could speak a little about the Merkle Root NEVILLE: Because I’ve not really heard too much about it. NEVILLE: Are you hashing every other transaction together? And then hashing those results? Yeah. I’m actually going to cover that in the next technical part, yeah. But just for now, as long as it makes sense that this Merkle Root provides you with a unique, short summary of all the transactions in the block And I’ll cover how that works and why a Merkle Root is used in a moment but good question. So there we have the Block Header. I haven’t covered all the block header data but this is the most important stuff So there we have a Block Header. The next step then Is to put this Block Header data in to a hash function in to the SHA256 hash function and then that will produce a random number as a result and this is called a “Block Hash” Any questions so far? Okay Okay, so that’s all well and good, but why would a miner do this? Or how does this help to get the block mined on to the blockchain? Well if this block hash, the result of putting this block header in to the hash function produces a block hash that is below a “Target” value Like so So let’s say this is the current Target value Which is a number internal to every Bitcoin node They all calculate their own… It’s like a shared Target value that is calculated in sync with every other node If they can get their block hash to be below this then the block will get added on to the blockchain and the block is solved, basically. So, so far with this first attempt this number here as you can see this target value starts with a certain number of zeros But this one is obviously a bigger number than that, so it’s not below the target So this first attempt at trying to add this block on to the blockchain has failed. What happens then, is that a miner doesn’t give up In the block header, there is space, there is an extra field called the “Nonce” which stands for a “Number used Once” This is like an open field that a miner is free to put their own number in, or data in. So what that means is, if this first attempt of the block header fails they can change the nonce value, or change the data in the nonce So for example, their first attempt might have had the number 0 in there then if that fails, they will change the nonce and then push the block header through the hash function again and because even though we’ve only made a small adjustment, like I showed you earlier with the Hash Function small adjustments can produce completely different results So a miner basically will construct a block header and keep changing this nonce value and hoping that the result will produce a block hash below the target value So literally, a miner; creates a block header, adds a nonce to the end, and if it doesn’t work they will keep incrementing it… hoping to get a low enough block hash Eventually if they’re lucky …this is just pure randomness and pure luck… They might be able to guess a nonce that works, that will produce a block hash below the target So here, this particular very high nonce has produced a successful block hash result. So there we are. What will happen then is, this node will add that block on to their blockchain and they will pass on this candidate block, this solved candidate block on to all the nodes they are connected to they will check it, and verify it, to make sure that the block header produces a block hash below the target they will add it to the file kick out any conflicting transactions, that are included in that block and then pass on the block, like so and they will do the same thing So that’s how Mining works on a basic level NEVILLE: So I had a quick question here NEVILLE: You’re saying that the Target value… NEVILLE: Each node has its own value for the Target, right? NEVILLE: So, there is a possibility that I mine a block, which is below my target value NEVILLE: And I give it to my neighboring nodes, and maybe the Target value has changed, or for some reason the other nodes don’t have the same Target value? NEVILLE: Is that a possibility? I understand your question. I think I explained the Target, not great… The “Target” Right… Basically the Target is there So that every 10 minutes a new block will be mined But if computers are hashing a lot faster then a block is likely to be mined in under 10 minutes so this Target value moves up and down so that a new block is mined every 10 minutes The target for the very first block, when you first start running Bitcoin It was hard-coded in at this value Then after every 2,016 blocks or roughly every two weeks What will happen is… Every node will look at the time it took for all the last 2,016 blocks to be mined The expected time for that is 20,160 minutes but if the actual time for all those blocks to be mined was less than that then the Target will adjust so the Target will adjust by this ratio So there we are So this ratio, because all the blocks were mined faster, is 0.9 And so this target, as you can see, will adjust downwards So every single node, when they receive a new block, after every 2,106 blocks they will all do this exact same calculation So every node started with this value but if everyone is on the same block then everyone will have calculated the same target So basically everyone shares the same target, they all calculate it themselves… But because they are all receiving the same blocks Everyone will calculate the same target value So not everyone has different target values, everyone has the same calculated target. So again, this happens every 2,106 blocks Same thing happens… how long did they take, what’s the ratio, okay that was mined faster… move it down again, like so So that’s how the Target works. Also, up until now, just to clarify… I’ve referred to this blockchain as a shared file or this shared file as a blockchain basically this shared file is full of transactions but because the transactions are added in blocks like so the file is called the “Blockchain” because each block builds on top of each other So with bitcoin it is a shared file, and this shared file is called the blockchain So if I just update the diagram… You can think of this shared file as blocks built on top of each other Finally… Why would anyone mine? What’s the incentive for a miner to use all this hashing power on their computer to try and mine the transactions from their memory pool on to the blockchain? Well, every time a miner mines a block, or is successful in mining a block they will receive what is called a “Block Reward” It started out at 50 But it halves roughly every 4 years So that’s the incentive to mine a block And the way a miner receives this block reward… When they construct the candidate block the very first transaction is reserved for them to put there So a Miner will put their own transaction at the top called a “Coinbase Transaction” And this is a special transaction because it allows them to send themselves a set amount of bitcoins that had not previously existed So if this block does get mined on to the blockchain they will be able to receive new bitcoins thanks to this coinbase transaction that they put there They put their bitcoin address in here so that they can receive the block reward. So just to summarize the basics of Mining A miner will get the transactions from the memory pool Then they’ll put their coinbase transaction at the top so that they can receive a block reward if they are successful in mining this block on to the blockchain Then they’ll create a block header Then they’ll hash it Hope that it’s below the target value If it’s not, they’ll keep adjusting the nonce over and over again and if they’re lucky they’ll find a nonce before anyone else that produces a block hash that is below the target. And then they have mined the block. Like so. Does that make sense? I think what I’ll do now, I’ll just show you a little Mining Simulator A little Mining Simulator So this is what a Bitcoin Miner might look like under the hood, basically. So what they’ll do, they’ll go in to their memory pool they’ll get the transactions from their memory pool This mining simulator is mining a block that has already been mined So this was a long time ago, there was only 13 transactions in the memory pool There’s obviously a lot more in there now What they’ll do then, they’ll construct the block header I haven’t covered these last two bits of the block header yet but don’t worry, I’ll cover them in a moment But this data refers to the previous block …has the merkle root, the version number And then what they’ll do, they will squish all of that data together in to one line of data and I’ve left some space here for the nonce So this is the block header data This is the target And then what a miner will do now, is they will take this block header and put a nonce in there keep incrementing it and these are the results, as you can see now… This is a very slow miner So as you can see, these are the nonce values on the left And these are the different hash values that that block header is producing NEVILLE: Okay, so when you created that string on top… NEVILLE: that did not include the transactions, it included only the header? Yes NEVILLE: Ohhh, okay. So the transactions I suppose are included in this Merkle Root This merkle root is the summary and refers to all the transactions in the block Also, here’s the code for this miner Just basically mines a block header Just hashes a block header I’ve made this code available on the website, I’ll give you a link to it in a moment But for this particular miner, I know what the successful hash val… grab this… there we are Let’s reduce this down by 100… So if I start this again I’ve increased that starting nonce value So again, this is the block header The nonce is going to start not from 0 this time but a much higher number And if I let it run now, hopefully after about 100 it will stop when it finds a block hash below the target value There we are Don’t know if you can see this… There we are, the block has been mined. So this particular block hash, purely random and just pure luck produced a block hash starting with this many zeros and that was below the target and so that’s all a miner does They just grab a block header change the nonce hash as fast as they can Does that make sense? I’ll put these links at the end. If you go on the website, on to I’ve put the mining simulator code here It’s written in Ruby, just because I think it’s a simple and easy to read language similar to Python I’ve tried to make this as simple as possible, so if you want to start, you know, hashing your own block headers you can On the website as well, on, if you go on to the browser You’ll be able to see, these latest blocks These are the the block headers And if you click on “Serialized” It will show you the serialized code And if you were to hash this, it would produce this block hash This is a Candidate Block basically it’s trying to do the same thing NEVILLE: Just curious, is this running off a machine that you have? NEVILLE: Is this live? Yeah this is live. This is a live candidate block, these are all live transactions It’s all on the browser So that’s the basics So that was the basics Now I’ll go on to the technical Basically I’m going to run through the exact same thing, but do it in a technical way So if you are interested in programming and actually mining a block This is the same stuff, but from a programmer’s perspective So again, just repetition going to repeat the same stuff Same thing, memory pool.. grab the transactions from the memory pool fill up your block There we are, this is the candidate block then you’ll put your coinbase transaction at the top The first transaction in a block is always a coinbase transaction that a miner has put there and so that allows you to claim the block reward which is currently 12.5 bitcoins The reason you would put the coinbase transaction at the top of the block after selecting all these transactions is because each of these transactions will have a fee and through your coinbase transaction you can collect all of these fees So instead of the coinbase transaction just receiving the 12.5 block reward you’ll be able to send yourself something like 13 or more depending on how much the fees are worth depends how much you can send yourself in this coinbase transaction NEVILLE: Okay so the coinbase transaction includes the fees? Yes NEVILLE: It’s not only the mining reward? Yes So you have the base level which is the block reward but then you can send yourself however much all these fees, from each transaction NEVILLE: So this means miners will tend to pick transactions with a higher fee? Exactly. Yes. NEVILLE: One more question on that. NEVILLE: I remember seeing a graph where it showed me on the left the fees and on the right how much time it will take to mine that block NEVILLE: So does this mean that if we put a very low fee, is it ever possible that our transaction will never get… never be part of a block? NEVILLE: Is that possible in your experience? Yes, what can happen is… say you make a transaction, it will go in to the memory pool first if there’s a lot of transactions in the memory pool then basically your transaction is competing with all these other transactions for space in a block Because a block used to be 1 megabyte’s worth of transactions It’s roughly around 1.7 now It uses a different metric called “weight”, but I’m not going to cover that just now But what will happen is, say you put such a low fee on there that it doesn’t make it in to a block, so it’s still hanging around in the memory pool if your transaction stays in the memory pool for longer than 72 hours …on average your transaction will just get kicked out of the memory pool So if you put a very low fee on a transaction there is a chance that it will never make it; 1. In to a candidate block …or even on to the blockchain NEVILLE: So again, in your experience, what’s the bare minimum… NEVILLE: Let’s say if I want to send five bucks to someone or buy a coffee, what’s the minimum fee that I have to pay? I think wallets generally tend to help you out with that but I don’t think they’re great at it What I would do is I would look at this candidate block and transactions in general are ordered in a candidate block from highest to lowest fees not strictly but I would head to the bottom of this candidate block and have a look at each of the fees on them You can basically just add a fee that is greater than the lowest fees That’s the simplest way to do it in my opinion NEVILLE: So again, this is per byte? I think when you hovered over… NEVILLE: So fee-per-byte… so this is… so the average byte size of a transaction? Is that what this is saying? Yes, so, this is a transaction this takes up space in a block then you set a fee on it but you’re basically paying for each individual byte so the bigger the transaction is data-wise the more of a higher fee you want to put on it A miner, they will select transactions based on fee-per-byte So which transactions are giving them the most fee for the space they are going to take up in the candidate block So again, just going to run through mining again… Put you coinbase transaction at the top Then you construct the block header same again, you have the version number This is an old block, the version number was 1 The way version numbers work now is different to how it used to be they can be used for voting I’m not going to cover that just now, I’m just going to run through a simple block header Then you have the previous block, same again, just refer to the block you want to build on top of which is basically the tip of the blockchain Then you have the merkle root So the way that works is… You have all the transactions in the block I’m just moving these to the side just for illustration purposes What you do is, you just run through every single pair of transactions and you hash them together Remember the hash function? So basically line these up, 1 and 2 together and hash them together and that will produce a “digest” or “hash result” And you do that for every single pair of transactions in the block If you end up with a transaction on its own that doesn’t have a pair All you do is you hash it with itself And that’s the first round done You have these 4 hashes from all these pairs And you just run through again hashing each pair together and you keep going on until you end up with a single hash result called the Merkle Root Now the question is If you want to create a summary of all the transactions in the block… (You’re hashing the Transaction IDs together here by the way) But if you want to create a summary or “fingerprint” for all the transactions in the block Why wouldn’t you just hash them all together, like this in a more simple way, why would you do it in this more elaborate tree structure? The reason for that is; say you were given a Merkle Root and you want to check if this Transaction ID is present in this block The only way you could check it would be to take all the other Transaction IDs you would need all the other ones including the one you want to check, and then hash them together to get the Merkle Root …to get the Hash, sorry. But with a Merkle Root Say you want to check the presence of this hash here all you actually need is; this one …to create this one and then if you’ve got that one then all you need is that one and then you can create that one and the all you need then is this one up here and that will give you this Merkle Root So the other one we did, was 1… 2… 3… 4… 5… 7, 6… transaction IDs But this one only needed 3. So obviously it doesn’t look like a massive improvement but this is just a simple block but if this was a longer block with thousands of transactions then you’re needing a lot less transaction IDs to check the presence of a single transaction I’ve not done this myself but apparently this is useful and efficient to do it this way. So that’s why we have the tree structure for hashing because it saves on checking for the presence of transactions in a block. NEVILLE: Just one question, when you say that we hash two transactions NEVILLE: You’re saying that we just concatenate them and hash… Yes, exactly. You just concatenate the transaction IDs NEVILLE: Which are basically the hashes of the transaction? *laughing* Yes So you basically hash transaction data to give you a transaction ID and then, yep… So there we go That’s the Merkle Root Time… also in the block header You just have the time which is a timestamp Like so But the way the time is encoded in the block header is using Unix Time Which is the number of seconds since 1 January 1970 So this block’s timestamp in a human representation is 12 May 2011 But in Unix Time that would be this many seconds A quick note on the Time These don’t have to be accurate They just have to be between a certain minimum and maximum value So the minimum value you can set as the time for your block header is the median time of the last 11 blocks or basically the time set in the block header of the sixth block down so you can’t have a time any lower than that that’s the minimum bound And then the maximum The maximum… Basically your node has its own time When you run it it just uses your computer’s time and then when other nodes connect to you they will send you what their time is on their computer and so you node will keep track of the difference between all of the times of the nodes you are connected to and if you put them in order and you take the median time of all the nodes you are connected to that’s called the Network Average Time So the maximum value of this Time is the network average time plus 2 hours. NEVILLE: Can you repeat this part quickly? NEVILLE: I just got a little confused, why is this time used, and is this the time to mine a block or something? Uh no, this time… I think this is… I think this is a bit unnecessary for me to explain Basically the Time… you just stick the current time of your computer in the block header But it doesn’t have to be precise So basically it just has to be between two certain values As long as it’s greater than the time of the sixth block down or the median time of the last 11 blocks it can’t be any lower than that because if you were to put a time lower than that then it would get rejected by the nodes you are sending it to And also the maximum time is basically the current time plus 2 hours. But network average time is used which is just the average time of all the nodes that are connected to you So don’t worry about that But all I’m saying is the time can be roughly an hour behind or 2 hours ahead So if a block comes through and it has a time before or ahead of a different block it doesn’t really matter it has this sort of flexible upper and lower boundary. Right, so that’s the Time. “Bits” Bits… remember the Target? Bits is just a compact representation of the Target So instead of putting the current Target the entire Target in the block header What you do… is you just grab the first 3 significant bytes or the first 6 significant characters after the leading zeros and then you put that there and then you just grab the offset so how far these three bytes are from the right and there are 26 bytes from the right here or in hexadecimal, 26 is “1a” So you just put that at the start there So Bits is just this compact representation of the Target Then that’s it then Then you just have the nonce, and you can set that to zero And so this is all the basic data now for the block header Do all these fields make sense? Okay good. Okay so now we’ve got all the fields ready to be mined We just have to get all the data in to the correct format So this is just a bit of programming to make sure everything is in the right order/format Each field should be in hexadecimal So, 1 is “1” in hexadecimal so this isn’t going to change But you just go through every single field making sure that it’s all hexadecimal So, these two are hexadecimal The Time… this is in decimal at the moment So we just convert that to hexadecimal And you do the same for every other field Secondly… Each field must be a fixed number of bytes So this has only got one character in here, but the Version field is 4 bytes So if we were to convert that in to 4 bytes it looks like this So 1 byte is 2 characters So “00”, that’s 1 byte And there we have 4 bytes in the Version Then you just go through every single field putting it in to the correct size so this is already 32 bytes that’s already 32 bytes The Time is going to be 4 bytes it’s already 4 Bits, same again… 4 And the Nonce then, that’s 4 bytes as well So for each field… make sure that you’ve padded it out with the right number of bytes So this basically means that a computer can read through each individual field easily. So that’s all the data in the right format and the right field size Lastly, the only slightly frustrating thing with Bitcoin is that when you send data to other nodes “across the wire” You have to send the data in reverse byte-order which is called “Network Byte Order” So obviously we have got all this in human-readable byte-order but when you hash and work with bitcoin data it has to be in reverse byte-order So you just go through every single field and put it in to network byte-order which is just reversing each byte So that’s something that tripped me up when I first started mining I was trying to hash these block headers and I was like “why am I not getting the same results?” It’s because, you want to get everything in network byte order. Just to show you, I’ve made a little quick… script… so, reversebytes So if I put “abcdef1234” And the way that works is; reverse byte order… you just split it all in to each individual byte so, 1 byte is 2 characters and then you just reverse the order of the bytes and that gives you that, so that’s how it works So you’re not just reversing the string you are reversing each byte. NEVILLE: So this might be a basic question, but… NEVILLE: So 2 characters is 1 byte? Yes That’s just something I’ve learnt. NEVILLE: I’m just looking at the C++… NEVILLE: I’ve not done C++ in a long time… NEVILLE: So, character type… size is 1 byte Umm Yeah, I don’t know that much deep about computer science All I know is… I’m doing this from more of a top down sort of level I’ve just learn’t that 2 character translates as 1 byte So there we are now we’ve got all the data finally in the right order all you do then is you concatenate each individual field together like that so this is the block header now it’s ready to put in to the hash function As I mentioned, SHA256 “Simple Hashing Algorithm 256” is used But in Bitcoin, everything gets hashed twice It’s just the way it was designed I don’t think it provides any massive benefits it’s just the way that Bitcoin hashes things so you just put this in to the hash function it spits out a result and then you hash the result to give you another result and that’s the final hash Quick note… When you hash things in Bitcoin this is in hexadecimal but you convert it in to binary first so you put binary data in to the hashing function and then that will spit out a binary result and you put the binary result in to the second hashing function that will spit out a second binary digest and then when you convert that second digest back in to hexadecimal that will give you the block hash But obviously this is in network byte-order again So if we reverse it, then that’s in the more human format So if you were to compare this to the Target you want to reverse back in to non-network byte-order So there we go I’ve only covered these things because they just trip you up Getting the fields is quite simple but then getting everything in to the right order and then converting it to binary and then switching it back again that’s what will trip you up if you are trying to mine a block header or hash a block header for yourself NEVILLE: Right so this was the block header on the left, which already had the fields in reverse network order Yes NEVILLE: Then you do a double hash of that Yep NEVILLE: And you convert it to hex and then you converted that again in reverse byte order I reversed it back NEVILLE: And this is… if you mined a block NEVILLE: This is the hash you would send on to the network? NEVILLE: Let’s say that you end up mining a block NEVILLE: And let’s say you get a nonce that is below the Target value NEVILLE: What do you do next? I’ll cover that now… So okay… You don’t actually send the block hash, you just send basically the block header and the transactions If you were programming you might not have to reverse it This is just so that I can display the results correctly In a human way So a miner, you just change the nonce Incrementing it; 0, 1, 2, 3, 4, 5… and so on There we are, obviously this is in network byte order and so on eventually you’ll get a hash that’s below the target if you’re lucky then that’s mined So when you’ve mined the actual block… So this is done now, this is a successful block header So if you send this block header to anyone else they can hash it for themselves and they’ll get the same result I think that’s what they call “Proof of Work” because you can just send this block header to anyone and they can prove for themselves that the hash is below the target So what you do then is, you send this block to all the nodes you are connected to and what you’re actually sending them is all of the transaction data concatenated together for each transaction before that you put a little transaction counter which is just a byte it’s a variable-sized byte I haven’t covered it, but it’s on the website it just tells you how many transactions are in the block and then before that you have the block header so this is what gets sent to all the nodes So you don’t have to send the block hash because they can calculate it for themselves So, that’s how mining works. What I thought I would do now is try and show you how you could mine an actual live block So, yeah… So if you run a bitcoin node you have the `bitcoin-cli` command which allows you to ask the bitcoin node questions one of them you can ask them is to `getblocktemplate` So what that will do, that will take your current memory pool and gather the highest value transactions that would fill up a block and give them to you So this is showing me all the transactions that would make a Candidate Block Let me just move this down… So, if I just quickly examine the information inside here If you use `bitcoin-cli help getblocktemplate` that will show you all the fields that it will produce But if I just quickly what we want to see… basically to mine a block is to create block header from all these transactions So `bitcoin-cli getblocktemplate` and I’m going to use “grep” just to search for the fields that we want I’m using `rg` which is a variant of `grep` And so we want to find, we want to have… A Version number We want the Previous Block Then we want the Merkle Root Then we want Time Then Bits Then there’s a Nonce, but we’re going to provide our own Nonce, so we don’t need that So looking at the… I spelled that wrong. There we are So looking at the results of `getblocktemplate` It’s giving us the version the Previous Block Hash It’s also giving us the Time current Time and Bits So just change that to current time So what it’s not giving us is a Merkle Root I don’t know why it’s not But what you can do What it is giving you is… This is just a JSON parser… It is giving you a list of transactions and it is giving you all the transaction IDs So if we just go through each one and say give us the transaction IDs Need to put that in quotes… There we are So these are all the transaction IDs in this candidate block So now we’ve got all those transaction IDs we just need to create our own Merkle Root for them So luckily I’ve written my own script that will take in a list of transactions and create a Merkle Root from them I’ve made this available on the website I’ll show it in a moment So if I just do that there we have a Merkle Root So now we’ve got all the data we need to mine a block or basically to construct a block header So if you just bear with me for five minutes I’m going to go through each field and yeah… So first of all we need to get the Version Oops, that’s, done wrong… So there we are Using `getblocktemplate` this is the current Version Like I said they’ve changed a bit They have been used for voting, so this is why it’s a very high number But I’m not going to explain it now, just take it as it is So this is the version number but obviously it’s in decimal So we need to convert that in to hexadecimal first So there we are And then now it’s in hexadecimal… This is already bytes So I just swap the byte order Or, swap the endianness because sometimes the byte order is referred to as “endianness”” So there we are So this is the version, all set up and ready to go So I’ll just trim the end off that because I’m going to concatenate it all together So if I save this as a variable So I’ll just `echo` that out to check So that’s the Version done. So next up after the Version is the… Previous Block So previous block hash… So this is the Block Hash for the Previous Block Let me just trim off the quotes And then… It’s already in hexadecimal It’s already 32 bytes So all I need to do is swap the byte order Or put it in to little-endian So there we are And then I can save this then as the `previousblock` variable Next up then is the Merkle Root Obviously we don’t get a Merkle Root from `getblocktemplate` command But what it does give us is the list of transactions and for each of those it’ll give us a transaction ID Oops Ah well I just lost that oh there it is 🙂 So there we go And from these I can get the Merkle Root There we are And then, it’s already in hexadecimal It’s already in 32 bytes Just need to swap the byte order again There we are, I’ll save that as a variable `merkleroot` Nearly there, I know it’s a bit laborious… So Version, Previous Block, Merkle Root, then we have the Time For some reason even though the `getblocktemplate` doesn’t give you the Merkle Root it does give you the current time on your computer Oh actually no it gives you a valid time I’m not sure if this is… I don’t know, but this is definitely a valid time But what you could do if you didn’t want to use that if you don’t want to use the `getblocktemplate` time You could just use the `date` function from a programming language So this gives you the number of seconds in Unix Time So I might just use that Just need to get it in to hexadecimal Like so 1, 2, 3, 4, 5, 6, 7… So that’s already 4 bytes Then we convert to reverse byte-order There we are, there it is, that’s the Time ready just trim the end off That’s the Time done Lastly “Bits” Which is the compact version of the Target There it is Just trim off the quotes and then put it in to reverse byte-order And there it is So now we’ve got all the basic block header data ready In the right format So if I… just construct the block header from each of those individual fields That is the… Version The Previous Block The Merkle Root Then… Time and then Bits So just check what that looks like There it is that’s our Block Header and so all we haven’t put there now is the Nonce So if we wanted to mine this we just need to create a loop that will keep hashing this block header and we’ll adjust the nonce, increment it, with every attempt So the Nonce will look like …it will start off with zero But obviously we need to get this in the right format So we need to convert that to NEVILLE: hexadecimal Thank you Like so But obviously this needs to be 4 bytes in length so I made a little function that will just make sure it’s the right size and then convert to swap endian Obviously it’s not very interesting because I’ve used the number zero But if I was to use a much higher Nonce value like that Like so, it’s prepared it for me So I’m not going to append this to it yet what I’m going to do; I’m going to create a loop So, `while true; do` We want to… Oh, wait there, before that I want to set a variable to increment Incremented variable ready So `while true; do` We want to create the Nonce which is going to be like I just showed you `echo $i` Then we want to get it in `dechex` hexadecimal 4 byte field size `swapendian` …trim off the end So that’s the nonce done. Then our attempt at trying to mine the block will be just the… Block Header we created plus the Nonce at the end And then what we want to do is we want to hash that all together So I’ve made a little function that will hash for me called `hash256` Because this is just the double hash; convert it to binary then double-hashes it We want to `hash256` the attempt And after every attempt we want to increment the nonce so `let “i++”` So just increment the nonce this number here And then `done` So hopefully this will work There we are, so now this is hashing very slowly a block header the current block header Obviously it’s very slow and it took me a long time but this is actually trying to mine a block So, if that was lucky what you’d probably do you’d have some code in there that would stop NEVILLE: If it reaches the target Target, yes And then what you’d do then there’s a function in bitcoin-cli called `submitblock` And you would just submit the header, the transaction count… and all the transaction data So if you constructed a successful Block Header you would send that to all the nodes you are connected to and you’d do that using the `submitblock` bitcoin-cli command NEVILLE: Is there a way to get the Target value from bitcoin-cli? Yeah, so… I think it’s included in bitcoin-cli `getblocktemplate` Yeah, so there it is This is the current Target. So you can grab that Target, and stick that somewhere in to the, yeah… in to the loop But obviously that hashing there was very, very slow NEVILLE: Oh yeah, I totally get that You’d probably want to do that in a single programming language This is just on Bash just for quick illustration purposes But that just goes to show how basically how simple mining is really The hardest part is just getting the data in the right order But after that you just increment a nonce and hash it and hope that you get a lucky result That’s everything I wanted to explain about how mining works That’s all the technical stuff but it’s probably a lot simpler than it’s made out to be On the website I’ve made all this code available so each of those little hexadecimal converters and decimal converters and the swap endianness they’re all here There is a Merkle Root function on here written in PHP It’s not the prettiest but it will take a list of Transaction IDs and create a Merkle Root for you You might not want to use it in production might want to use a different library I’m sure there’s loads of Bitcoin libraries out there with their own Merkle Root functions for you to use Is there any questions about that mining process? NEVILLE: No, this was really awesome Greg, thanks a lot. NEVILLE: Just curious, any experience with those heavy-duty miners? No, I’ve not actually mined myself because, like we said, it’s so inefficient to mine on your laptop if you wanted to mine now it’s not a hobby thing, you have to… buy a professional mining rig and run it like a proper business which is a shame because that takes mining out of the hands of the average person then So I think if you wanted to get in to mining you’d have to really commit to it NEVILLE: Even those guys would be doing, like at a fundamental level they would be doing the same thing? NEVILLE: Just faster? Yes So they’re just doing the exact same thing Constructing a Block Header and just hashing it as fast as you can This whole construction of the block header is simple and takes an absolute fraction of a second to do and then the whole Mining then the whole meat of mining is hashing as fast as possible Don’t think I’ve missed anything I think I’ve covered everything I wanted to I know that was probably a bit repetitive in parts NEVILLE: Oh no no, this was a great explanation again, thanks a lot for this But I think if you’re starting out with Bitcoin one of the best ways to learn, I think, is to try and mine your own Block Header I think it’s quite simple to do And I think mining is one of the first fundamental parts of how Bitcoin works so if you can figure out how to mine a Block Header or hash a Block Header that’s a great stepping stone That will give you the tools then to figure out about all the hexadecimal, and network byte order, and how to hash and you can take those tools then and use them in other parts of Bitcoin like in Transactions and all that So this would be my first place to start if you were to try coding with Bitcoin You’re probably not going to set up your own miner but as a learning experience this is a great place to start So if you want to browse block headers just go on to This is the blockchain And you can browse through each individual block And see all the different block headers and test for yourself that you can get the same block hash results NEVILLE: Nice So, thank you very much for that That was good to do NEVILLE: Thanks No worries.


15 thoughts on “Bitcoin Lesson | Mining”

  • Great video… Keep up the great work… Just one note the part about not knowing why 1 byte is 2 characters is such a fail… Give you a hint… Try to convert 255 to hexadecimal…. Tada… If you would write each byte in decimal it would be 000 to 255… 3 characters per byte… With hexa you only need 2 – 00-FF. 1 character per byte is impractical because you would need 255 different symbols for each number. Additionally if you convert hexa to binary you will see additional advantage… First character corresponds to first 4 bits and second characters to second 4 bits in a byte…. Example 0x11 in binary is 00010001… 0x55 would be 01010101… You see how nicely it aligns? One hexa number 0-F is decimal 0-15 which is exactly what you can store in 4bits. Binary 1111 is decimal 15 and hexadecimal F. Binary 11111111 (highest number storable in byte) is decimal 255 which is FF in hexa. Hexadecimal means 16 based which is power of 2. Computer understands and works in binary numbers of 8bits (bytes) or multiples of 8 bits. For us humans best way to write such numbers (compact and easy to understand and navigate) is hexadecimal. Hexadecimal is just most practical way for humans to write arbitrary bytes (seen memory dump for example?) using alphabet+numbers. Reason for that being binary and hexadecimal are both powers of 2 while decimal isn't

  • Awesome Video man! Was searching for exactly this kind of video. Really helped me out. Please keep uploading crypto related stuff 🙂

  • Thank you very much for this video. I have cleared all of my concepts how block is mined and work behind it. I just want to answer the question which was asked at 49:31 that how could be 1 Byte is equal to 2 Character as in programming language C++ it should be 1 character and 1 character is equal to 8 bits. My answer is because here number are in Hexadecimal format which represent 2 character FF in one byte and in binary it represent 8 Bits 1111 111. Both FF and 1111 1111 are equal. If we convert FF to binary the answer will be 1111 1111 bits. Hope i answered the question at the best of my ability.

  • Sajjad Munawar says:

    @learnmeabitcoin I have some questions that when does miner or pool of miners start the processing of POW when all the Transactions are completed in candidate block? Then they keep repeating the POW by changing the nonce to be under thr difficulty target. If so then how they decided to stop adding other transactions in candidate block. What is the criteria? If they stopped adding transactions then do these get accumulate in mem pool untill miners do not complete the mining process?

  • Hello, At 1:06:12 when trying to submit block along with transactions, Does it need to verify all transactions to be valid or getblocktemplate() give us already verified blocks ?

  • manish maharjan says:

    Is there a limitation of no. of txns to form the candidate block ? Can we put any no. of transactions depending upon txn fee to mine the block ?

  • Question 1: What happens if two miners both found correct results and start to propagate in the network for the blocks which have contradict transactions (like spending the same coin twice) , which one gets selected? (If this ever has a chance to happen)

    Question 2: So if you need to verify if a transaction is in a block, you need to hashing the transaction id with some other hashed transaction ids then do the comparison. How do you get the hashed transaction ids on the first place? Don't you need to hash other transaction ids first? If that's the case, we then ended up with doing more hashing. However, if they save the hashed transactions' ids in a file, that would make sense, but more space used?

  • The part of the target is the most confusing to me. So is there a new target for every block that is added to the blockchain?? The objetive of the hash is to find the number below the target, right?

  • Vinicius Lampert Ferrari says:

    I have a question about Blockchain. The person who chooses the transactions in the block is the same person who mines the block, or is it possible for one person to select the transactions in the block and then another miner solves the hash?

Leave a Reply

Your email address will not be published. Required fields are marked *