How can we call the function of the Custom Controller from the Javascript of VisualForce Page

You can call the function of the Custom Controller from the Javascript using action function. Here is a simple example

VF page

<apex:page controller="myController">

<script>
function callJavaScript(){
callAction();
}
</script>

<apex:actionFunction action="{!myMethod}" name="callAction" reRender="none"/>
<button onclick="callJavaScript();">Call Action Function</button>

</apex:page>

Controller

public class myController{

public myController(){}

public void myMethod(){
system.debug('Method called');
}
}