Answer:
[tex]\displaystyle \cos (G)=\frac{36}{85}[/tex]
Step-by-step explanation:
Recall that cosine is the ratio of the adjacent side to the hypotenuse:
[tex]\displaystyle \cos(G)=\frac{\text{adjacent}}{\text{hypotenuse}}[/tex]
The hypotenuse is always the side opposite to the right angle. In this case, it is 
85. 
The adjacent side to G is 36. 
Therefore: 
[tex]\displaystyle \cos (G)=\frac{36}{85}[/tex]
For your other question, remember that in a right triangle, cosine is the ratio of the adjacent side to the hypotenuse. 
As you know, the hypotenuse is always the longest side in a right triangle. 
Therefore, the ratio of the adjacent side to the hypotenuse will never be greater than 1. 
Since 85/77 is greater than 1, it is undefined. 
The same applies to sine as well.