Although the Megaminx has twice as many faces as a Rubik's cube, there are usually only three adjacent faces used in one algorithm, which allows 3x3x3 notation to be used. There are some algorithms that require four or five adjacent faces that are not related in a way where 3x3x3 notation could be used. Thus, extra notation (or an explanation of how the normal notation works) is needed to write these algorithms without confusion.
The official notation used by the WCA is called Pochmann notation, which was invented by Stefan Pochmann in 2007. It uses only six kinds of moves: U and U', which are executed exactly as they are in 3x3 notation, and R++, R--, D++, and D--. To execute a move, hold the Megaminx by the L face (for an R turn) or the U face (for a D turn) with the left hand use the right hand to move the bulk of the puzzle two-fifths of a turn i the indicated direction. For example, R++ means you should hold the L face and rotate the bulk of the puzzle two-fifths of a rotation (144 degrees) clockwise.
Carrot notation was invented by Oscar Roth Andersen (nickname "Carrot") in 2014 as an improvement over Pochmann notation. Its aim is to improve readability and decrease length so the puzzle can be scrambled more accurately. In order to achieve this, U and U' moves remain unchanged while R and D moves are condensed into a sequence of two letters so that Rxx Dyy in Pochmann notation (where x and y can be either + or -) is reduced to xy in Carrot notation. For example, R++ D++ R-- D++ would equal ++ -+ in Carrot notation. Additionally, a line break is added after U and U' moves while odd lines are pushed slightly to the right (usually done by starting the odd line with a white space) to prevent the scrambler from accidentally going to the wrong line.
The following moves in Pochmann notation
R-- D++ R++ D++ R++ D++ R++ D++ R++ D-- U' R++ D-- R-- D++ R++ D-- R++ D-- R-- D++ U R++ D-- R-- D-- R++ D-- R-- D++ R-- D-- U R++ D-- R-- D++ R++ D++ R-- D++ R-- D++ U' R-- D++ R-- D-- R++ D-- R-- D-- R-- D++ U' R++ D++ R++ D-- R-- D-- R++ D++ R++ D-- U' R-- D++ R++ D++ R-- D++ R-- D-- R-- D-- U
equal these in Carrot notation
-+ ++ ++ ++ +- U' +- -+ +- +- -+ U +- -- +- -+ +- U +- -+ ++ -+ -+ U' -+ -- +- -- -+ U' ++ +- -- ++ +- U' -+ ++ -+ -- -- U
Despite the reduced length and increased readability, Carrot notation is not widely used since Pochmann notation is well established and more intuitive for beginners. However, proposals have been made to use Carrot notation officially. 
The Backright and Backleft Faces
When solving the last layer, the Megaminx has five faces that surround the top face while the 3x3x3 has four. This means that two of those faces on the Megaminx are in the spot of one face on the 3x3x3. The one face on the 3x3x3 is commonly the back face (B), and the two faces on the Megaminx that are in its spot are the backright (bR) and backleft (bL) faces.
Here is an example of an algorithm that requires the backright and backleft faces:
R bR bL U bL' U bR' U2' R'
This algorithm orients all of the edges on the top face of the last layer when one edge is already oriented (and is the closest edge to you). A clockwise turn of the right face is followed by a clockwise turn of the face that is next to it, which is followed by a clockwise turn of the face that is next to that one.
Notice that there is a U2' in this algorithm. On a 3x3x3, a U2' would be the same as a U2, making it useless unless you are specifying finger tricks. On a Megaminx, a U2' is different from a U2. A U3' would be the same as a U2, and a U4' would be the same as a U.
Cube rotations on the Megaminx are fairly self-explanatory. y (or [u]) and z (or [f]) are performed as normal; however, x (or [r]) requires you to look at the algorithm to decide how to perform it. If the algorithm contains mostly R moves, the right face should probably remain the right face after an x rotation and the backright face should probably become the up face after an x' rotation. If the algorithm contains mostly L moves, the left face should probably remain the left face after an x rotation and the backleft face should probably become the up face after an x' rotation.
Lowercase letters (or a big letter followed by a w) on a 3x3x3 indicate a turn of that face and any middle layers there might be on that puzzle. It is hard to explain how this works on a Megaminx, so here is an example:
r' (or Rw') on a 3x3x3 would be a turn of everything except the left face, which would result in the back face becoming the up face. On a Megaminx, it would also be a turn of everything except the left face, which would result in the backleft face becoming the up face.